Systems, environment and methods for evaluation and management of autism spectrum disorder using a wearable data collection device

ABSTRACT

The systems, environment, and methods, described herein support evaluation of an individual for ASD while in the home environment. Through data collected by a wearable data collection device donned by the individual, eye contact with the caregiver, verbal interaction, and repetitive verbalizations and motions of the head and body may be tracked and objectively quantified during evaluation. Further, the wearable data collection device may support monitoring of brain activity and other physiology which, in turn, may be analyzed by the systems and environment described herein during evaluation to recognize patterns that predict evaluation outcome and other clinical features. Various software modules and tools supported by the wearable data collection device provide training, ongoing progress tracking, and management solutions for individuals living with ASD.

RELATED APPLICATIONS

The present application is related to and claims the priority of U.S.Provisional Application No. 61/888,531 entitled “A Method and Device toProvide Information Regarding Autism Spectrum Disorders” and filed Oct.9, 2013, and U.S. Provisional Application No. 61/943,727 entitled“Method, System, and Wearable Data Collection Device for Evaluation andManagement of Autism Spectrum Disorder” and filed Feb. 24, 2014, thecontents of each of which are hereby incorporated by reference in theirentireties.

BACKGROUND

Autism probably begins in utero, and can be diagnosed at 4-6 months.However, right now in America, Autism is most often diagnosed at 4-6years. The median diagnosis age in children with only 7 of the 12classic Autism Spectrum Disorder symptoms is over 8. In these missedyears, the child falls much further behind his or her peers thannecessary. This tragedy is widespread, given that 1 in 42 boys isestimated to have Autism (1 in 68 children overall) (based upon U.S.Centers for Disease Control and Prevention, surveillance year 2010).Additionally there are few methods of managing or treating Autism, andalmost no disease-modifying medical treatments. Why do these diagnosisand treatment gaps exist?

There is no blood test for autism. Nor is there a genetic, neural orphysiological test. Astonishingly, the only way parents can know iftheir child has autism is to secure an appointment with multiple doctors(pediatrician, speech pathologist, perhaps neurologist) who observe thechild playing and interacting with others, especially with thecaregiver. This is time-consuming, must be done during doctors' hours,is challenging and contains subjective components, varies by clinician,does not usually generate numerical data or closely quantified symptomsor behaviors; and demands resources, knowledge and access to the healthsystem—all contributing to delayed diagnosis.

There are also social factors. A parent's suspicion that his/her childhas autism generally takes time to grow, especially with the first childor in parents with little child experience (no frame of reference).Furthermore, the decision to seek help may be clouded by fear, doubt,denial, guilt, stigma, embarrassment, lack of knowledge, distrust of themedical system, and confusion. Once the decision is made, it can be aprotracted, uphill battle to find the right care center and secure thescreening appointment and a correct diagnosis. All these factors areamplified for at-risk families with low SES, low education level,language and cultural barriers, familial ASD; and in single-parent ordual job families. Time that passes before diagnosis reduces the child'ssocial and emotional development, learning of language, and eventuallevel of function in society.

Even if the family surmounts various hurdles and comes in for anofficial diagnosis, hospital admission and the test environment can bedaunting and unnatural, especially for those with language, cultural orSES barriers.

In this context, a shy child may seem autistic and an ASD child maycompletely shut down, especially since ASD children are particularlyaverse to changes in familiar settings and routines. Thus, the child maybe diagnosed as further along the Autism spectrum than is the reality,and false diagnoses such as retardation may be attached. This hasprofound consequences in terms of what schooling options are availableto the child, how the parents and community treats the child, and therelationship that gets set up between the parents and the healthcaresystem. Even in a friendly testing lab, clinicians cannot see the childplay and interact exactly as he/she does in the familiar homeenvironment, and can never see the child through the caregiver's eyes,nor see the world through the child's eyes. Importantly, there are nowidely adopted systems for objectively quantifying behavioral markersnor neural signals associated with ASD, especially at home.

Even when and if a diagnosis is achieved, there are few optionsavailable to the family (or to the school or health care giver) thatquantify the degree of severity of the child's symptoms. Autism is aspectrum of course, and people with autism spectrum disorders have arange of characteristic symptoms and features, each to varying degreesof severity if at all. Measuring these and characterizing the overalldisorder fingerprint for each person is an important advance for theinitial characterization, as per above, but importantly this fingerprintis dynamic over time, especially in the context of attempted treatmentsand schooling options, so measuring the changing severity and nature ofeach feature is important. This is the tracking or progress-assessmentframework. Additionally, perhaps one of the greatest unmet needs withinASD comes in terms of the treatment or training framework. That is tosay, mechanisms for providing intervention of one kind or another thatcan have a disease-modifying or symptom-modifying impact. There are fewoptions available to the families affected, and again, there are fewoptions for rigorously quantifying the results.

SUMMARY

Ideally, a child would be assessed for ASD in the family's regular homeenvironment, on their schedule, in their language, and perhaps whileallowing remote doctors to literally look through the caregiver's eyesat the child (and vice versa). It should be private and confidential,quick and convenient, quantitative and repeatable, and low-cost enoughthat a worried parent will pay the cost directly (thus bypassing thecomplexity of insurance reimbursements). In some implementations, thesolution tracks and objectively quantifies clinical features such aswhen the child directs his gaze toward the caregiver's voice, howfrequently he interacts with the caregiver verbally, and repetitiveverbalizations and motions of his head and body. The solution, in someimplementations, records brain activity and other physiology, andapplies algorithms to recognize patterns that predict diagnosis andclinical features. The algorithmic analysis, for example, may beconducted in a central (e.g., cloud-based) system. Data uploaded to thecloud can be archived and collected, such that learning algorithmsrefine analysis based upon the collective data set of all patients. Insome implementations, the system combines quantified clinical featuresand physiology to aid in diagnosing autism objectively, early, and atleast semi-automatically based upon collected data.

In some implementations, an evaluator reviews a recorded sessionuploaded to the central system and makes a diagnosis. Evaluators, insome implementations, may perform a live (supervised) session, or reviewanother clinician's live session, through real-time data feed betweenthe family home session and a remote evaluator computing system.Although described as an in-home system due to the advantages describedabove, the system may additionally be used within a clinical environmentto aid in evaluation of an individual.

The system has several advantages. Evaluation can be performed in thehome, in an organic and comfortable play-space, in the family'slanguage, whenever the caregiver has time, discreetly, privately,extremely rapidly (kits can arrive same-day, and testing complete in afew hours), and inexpensively. The most costly equipment, the wearabledata collection device, can be reused, so the main costs are evaluationkit objects (e.g., toys or other interaction objects) andevaluator/technician time.

The system is beneficial to evaluators as well. Evaluators get access tomany more subjects. The evaluators can perform diagnosis from home,during commute, or otherwise away from the office. Evaluators areafforded the opportunity to observe patients in their naturalenvironment, and can witness transient behavioral events that otherwiseonly caregivers might see. If evaluators are working as a team to reviewan individual, they do not have to match their schedules to be in oneplace, but can jointly observe a single session with the individual fromas many locations as there are team members. Above all, the system coulddecrease the age of Autism diagnosis drastically, and reach many at-riskfamilies, early.

In some implementations, the system goes beyond the evaluation stage totrack an individual's ongoing progress. For example, after diagnose ofASD, there is typically a long series of interventions from schools,doctors, etc. At some point, the child either does or does not developsocially and academically to a level where she can function in society.In between, there is rarely a point for interim evaluation andassessment to gauge progress. Maybe, a few years down the road, thefamily will have a follow-up “diagnosis” appointment. However, thefollow-up visit will likely involve a different set of professionals,leaving the evaluation open to vagueness. Some programs for tracking ASDprogress exist, having set goals and milestones, but they too can bevague and infrequently assessed. In employing a system such as thewearable data collection system described above for ongoing tracking ofbehaviors, abilities, and functionality of an ASD diagnosed child, afamily can benefit from an exacting, quantitative-by-nature, cheap,at-home, understandable, standardized, comparable (from one time pointto another), numerical assessment of a child's individualcharacteristics. The system, for example, could provide high-frequency(e.g., up to daily) assessments, each with perhaps hundreds or thousandsor more data points or samples such as (in)correct behaviors or relevantbrain states, which can be incorporated into the child's everyday homelife to measure the child's ongoing progress.

To enable such ongoing assessment, in some implementations, applicationsfor assessment as the child's development progresses may be madeavailable for download to or streaming on a wearable data collectiondevice via a network-accessible content store such as iTunes or Playstore, or YouTube or other content repositories, or other contentcollections. Content providers, in some examples, can include educators,clinicians, physicians, and/or parents supplied with developmentabilities to build new modules for execution on the wearable datacollection device evaluation and progress tracking system. Content canrange in nature from simple text, images, or video content or the like,to fully elaborated software applications (“apps”) or app suites.Content can be stand-alone, can be playable on a wearabledata-collection device based on its existing capabilities to playcontent (such as in-built ability to display text, images, videos, apps,etc., and to collect data), or can be played or deployed within acontent-enabling framework or platform application that is designed toincorporate content from content providers. Content consumers,furthermore, can include individuals diagnosed with ASD or theirfamilies as well as clinicians, physicians, and/or educators who wish toincorporate system modules into their professional practices.

In some implementations, in addition to assessment, one or more modulesof the system provide training mechanisms for supporting theindividual's coping and development with ASD and its characteristicssuch as, in some examples, training mechanisms to assist in recognitionof emotional states of others, social eye contact, language learning,language use and motivation for instance in social contexts, identifyingsocially relevant events and acting on them appropriately, regulatingvocalizations, regulating overt inappropriate behaviors and acting-out,regulating temper and mood, regulating stimming and similar behaviors,coping with sensory input and aversive sensory feelings such asoverload, and among several other things, the learning of abstractcategories.

In autism as well as other conditions and in healthy individuals, it isadvantageous to measure heart rate and other physiological signals,especially when measuring behaviors associated with a condition. Forinstance if a child looks at an adult who is displaying a facialexpression of anger and the child's heart rate quickens, that is anindication that some aspect of the emotional expression was processed.If, however, a child looks at the same adult and experiences no changein heart rate or dynamics, this is consistent with being unaware of orinsensitive to the emotional expression, which is more consistent withthe response of a child with ASD.

In one aspect, the present disclosure relates to a system and methodsfor inexpensive, non-invasive measuring and monitoring heart rate andcardiovascular dynamics using a wearable data collection device throughanalysis of a variety of motion sensor data. It is advantageous to beable to measure heart rate and cardiovascular dynamics as non-invasivelyas possible. For instance, the ability to avoid electrodes, especiallyelectrodes that must be adhered or otherwise attached to the skin, is inmost situations preferable, particularly for children who do not likeextraneous sensory stimulus on their skin. It is also advantageous to beable to derive, from a non-invasive signal, additional cardiovasculardynamics beyond simply heart rate, such as dynamics that may indicateunwellness and which may usually require multi-lead ECG setups andcomplex analysis.

In some implementations, a wearable data collection device including oneor more motion sensors capable of discerning small motions of the bodyis placed comfortably and removably on an individual without need forgels or adhesives. The wearable data collection device may be a devicespecifically designed to measure and monitor cardiovascular dynamics ofthe body or a more general purpose personal wearable computing devicecapable of executing a software application for analyzing small motiondata to obtain cardiovascular dynamics data.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1A is a block diagram of an example environment for evaluating anindividual for Autism Spectrum Disorder using a wearable data collectiondevice;

FIG. 1B is a block diagram of an example system for evaluation andtraining of an individual using a wearable data collection device;

FIGS. 2A and 2B are a swim lane diagram of an example method forperforming a remote evaluation of an individual using a wearable datacollection device;

FIG. 3A is a block diagram of an example computing system for trainingand feedback software modules incorporating data derived by a wearabledata collection device;

FIG. 3B is a block diagram of an example computing system for analyzingand statistically learning from data collected through wearable datacollection devices;

FIG. 4 is a flow chart of an example method for conducting an evaluationsession using a wearable data collection device donned by a caregiver ofan individual being evaluated for Autism Spectrum Disorder;

FIG. 5A is a block diagram of an example environment for augmentedreality learning using a wearable data collection device;

FIG. 5B is a block diagram of an example collection of softwarealgorithms or modules for implementing language and communication skilltraining, assessment, and coaching using a wearable data collectiondevice;

FIG. 5C is a screen shot of an example display for coaching a user inperforming a bow;

FIG. 5D is a screen shot of an example display for providingconversation skill feedback to a user;

FIG. 6A through 6D illustrate a flow chart of an example method foraugmented reality learning using a wearable data collection device;

FIGS. 7A through 7C illustrate a flow chart of an example method foridentifying socially relevant events and collecting informationregarding the response of an individual to socially relevant events;

FIG. 7D illustrates a screen shot of an example feedback display forsuggesting an intervention to a user;

FIG. 8 is a flow chart of an example method for conditioning social eyecontact response through augmented reality using a wearable datacollection device;

FIG. 9 is a block diagram of an example collection of softwarealgorithms for implementing identification of and gauging reaction tosocially relevant events;

FIG. 10A is a flow chart of an example method for identifying andpresenting information regarding emotional states of individuals near anindividual;

FIGS. 10B and 10C are screen shots of example user interfaces foridentifying and presenting information regarding emotional states of anindividual based upon facial expression;

FIG. 11 is a block diagram of an example system for identifying andanalyzing circumstances surrounding adverse health events and/oratypical behavioral episodes and for learning potential triggersthereof;

FIG. 12 is a block diagram of an example wearable computing device; and

FIG. 13 is a block diagram of an example computing system.

DETAILED DESCRIPTION

As illustrated in FIG. 1A, an environment 100 for evaluating anindividual 102 for autism spectrum disorder includes a wearable datacollection device 104 worn by the individual 102 and/or a wearable datacollection device 108 worn by a caregiver 106, such that data 116related to the interactions between the individual 102 and the caregiver108 are recorded by at least one wearable data collection device 104,108 and uploaded to a network 110 for analysis, archival, and/orreal-time sharing with a remotely located evaluator 114. In this manner,evaluation activities, to be evaluated in real time or after the fact bythe evaluator 114, may be conducted in the individual's accustomedsurroundings without the stress and intimidation of the evaluator 114being present. For example, evaluation activities may be conducted in afamily's home environment at a time convenient for the family members.

Evaluation activities, in some implementations, include a set of playsession phases incorporating, for example, various objects forencouraging interaction between the caregiver 106 and the individual102. For example, the caregiver 106 may be supplied with an evaluationkit including one or both of the individual's data collection device104, the caregiver data collection device 108, a set of interactiveobjects, and instructions on how to conduct the session. The set ofinteractive objects, in one example, may include items similar to thoseincluded within the Screening Tool for Autism in Toddlers (STAT™) testkit developed by the Vanderbilt University Center for TechnologyTransfer & Commercialization of Nashville, Tenn. The instructions, inone example, may be provided textually, either online or in a bookletsupplied in the evaluation kit. In another example, the instructions arepresented in video form, either online or in a video recording (e.g.,DVD) included in the kit.

In some implementations, the instructions are supplied via the caregiverwearable data collection device 108. For example, the wearable datacollection device 108 may include an optical head-mounted display (OHMD)such that the caregiver may review written and/or video instructionsafter donning the wearable data collection device 108. The caregiver mayperform a play session or test session based on the instructions, or bymirroring or responding to step-by-step directions supplied by a remoteevaluator 114, who can be a trained clinician or autism specialist, suchthat the remote evaluator 114 can walk the caregiver 106 through theprocess step by step, and the remote evaluator 114 can observe andevaluate the process and the behaviors of the individual 102 and otherdata in real time and directly through the eyes of the caregiver 106(via a camera feed from the data collection device 104).

The wearable data collection device 104 or 108, in some implementations,is a head-mounted wearable computer. For example, the wearable datacollection device 104 or 108 may be a standard or modified form ofGoogle Glass™ by Google Inc. of Mountain View, Calif. In other examples,the wearable data collection device 104 or 108 is mounted in a hat,headband, tiara, or other accessory worn on the head. The caregiver 108may use a different style of data collection device 108 than theindividual 102. For example, a caregiver may use a glasses stylewearable data collection device 108, while the subject uses ahead-mounted visor style of data collection device 104.

In some implementations, the data collection device 104 for theindividual 102 and/or the data collection device 108 for the caregiver106 is be composed of multiple portions 105 of body-mountable elementsconfigured to mount on different areas of the body. In general, thewearable data collection device 104 or 108 may be configured as asingle, physically-contiguous device, or as a collection of two or moreunits that can be physically independent or semi-independent of eachother but function as a whole as a wearable data collection device 104or 108. For example, the data collection device 104 or 108 may have afirst portion including an optical head-mounted display (OHMD) and whichtherefore is mounted on or about the head such as in a modified versionof eyeglasses or on a visor, hat, headband, tiara or other accessoryworn on the head. Further, the data collection device 104 or 108 mayhave a second portion separate from the first portion configured formounting elsewhere on the head or elsewhere on the body. The secondportion can contain, in some examples, sensors, power sources,computational components, data and power transmission apparatuses, andother components. For instance, in an illustrative example, the firstportion of data collection device 104 or 108 may be used to displayinformation to the user and/or perform various tasks of user interface,whereas the second portion of data collection device 104 or 108 may beconfigured to perform sensing operations that are best suited tospecific parts of the body, and/or may be configured to performcomputation and in so doing may consume power all of which may require asize and bulk that is better suited to be elsewhere on the body than ahead-mounted device. Further to the example, the second portion of datacollection device 104 or 108 may be configured to mount on the wrist orforearm of the wearer. In a particular configuration, the second portionmay have a design similar to a watch band, where the second portion canbe interchanged with that of a standard-sized wrist watch and therebyconvert an off-the-shelf wrist watch into a part of a smart ecosystemand furthermore hide the presence of the second portion of the datacollection device 104 or 108. Although described as having two portions,in other implementations, the wearable data collection device 104 or 108may include three or more portions physically independent of each otherwith each portion capable of inter-communicating with at least one ofthe other portions. Many other configurations are also anticipated.

The wearable data collection device 104 for the subject may becustomized for use by an individual, for instance by making it fit thehead better of someone of the age and size of a given individual 102, orby modifying the dynamics of the display such that it is minimallydistracting for the individual 102. Another possible customization ofthe wearable data collection device 104 includes regulating the amountof time that the wearable data collection device 104 can be used so asto cause minimal change to the individual 102, such as to the developingvisual system of the individual 102. The wearable data collection device104, in a further example, may be customized for the individual 102 tomake the wearable data collection device 104 palatable or desirable tobe worn by the individual 102 for instance by cosmetic or sensorymodifications of the wearable data collection device 104.

The wearable data collection device 104 or 108, in some implementations,can be modified for the type of usage discussed herein, for instance byequipping it with an extended-life power source or by equipping it withan extended capacity for data acquisition such as video data acquisitionwith features such as extended memory storage or data streamingcapabilities, or the like.

Rather than performing the described functionality entirely via awearable data collection device 104 or 108, in some implementations, thedata collection device 104 or 108 includes a bionic contact lens. Forexample, the OHMD may be replaced with a bionic contact lens capable ofproviding augmented reality functionality. In another example, animplantable device, such as a visual prosthesis (e.g., bionic eye) mayprovide augmented reality functionality.

The wearable data collection device 104 or 108 can be arranged on thebody, near the body, or embedded within the body, in part or entirely.When one or more components of the wearable data collection device 104or 108 is embedded within the body, the one or more components can beembedded beneath the skin; within the brain; in contact with input oroutput structures of the body such as peripheral nerves, cranial nerves,ganglia, or the spinal cord; within deep tissue such as muscles ororgans; within body cavities; between organs; in the blood; in otherfluid or circulatory systems; inside cells; between cells (such as inthe interstitial space); or in any other manner arranged in a way thatis embedded within the body, permanently or temporarily. When one ormore components of the wearable data collection device 104 or 108 isembedded within the body, the one or more components may be insertedinto the body surgically, by ingestion, by absorption, via a livingvector, by injection, or other means. When one or more components of thewearable data collection device 104 or 108 is embedded within the body,the one or more components may include data collection sensors placed indirect contact with tissues or systems that generate discernible signalswithin the body, or stimulator units that can directly stimulate tissueor organs or systems that can be modulated by stimulation. Datacollection sensors and stimulator units are described in greater detailin relation to FIG. 12.

The wearable data collection device 104 or 108 can be configured tocollect a variety of data 116. For example, a microphone device builtinto the data collection device 104 or 108 may collect voice recordingdata 116 a, while a video camera device built into the data collectiondevice 104 or 108 may collect video recording data 116 b. The voicerecording data 116 a and video recording data 116 b, for example, may bestreamed via the network 110 to an evaluator computing device(illustrated as a display 112) so that the evaluator 114 reviewsinteractions between the individual 102 and the caregiver 108 inreal-time. For example, as illustrated on the display 112, the evaluatoris reviewing video recording data 116 j recorded by the caregiverwearable data collection device 108. Additionally, the evaluator may belistening to voice recording data 116 a.

Furthermore, in some implementations, the wearable data collectiondevice 104 is configured to collect a variety of data regarding themovements and behaviors of the individual 102 during the evaluationsession. For example, the wearable data collection device 104 mayinclude motion detecting devices, such as one or more gyroscopes,accelerometers, global positioning system, and/or magnetometers used tocollect motion tracking data 116 h regarding motions of the individual102 and/or head position data 116 d regarding motion particular to theindividual's head. The motion tracking data 116 h, for example, maytrack the individual's movements throughout the room during theevaluation session, while the head position data 116 d may track headorientation. In another example, the motion tracking data 116 h maycollect data to identify repetitive motions, such as jerking, jumping,flinching, first clenching, hand flapping, or other repetitiveself-stimulating (“stimming”) behaviors.

In some implementations, the wearable data collection device 104 isconfigured to collect eye tracking data 116 g. For example, the wearabledata collection device 104 may include an eye tracking module configuredto identify when the individual 102 is looking straight ahead (forexample, through the glasses style wearable data collection device 104)and when the individual 102 is peering up, down, or off to one side. Inanother example, the individual's data collection device 104 isconfigured to communicate with the caregiver data collection device 108,such that the wearable data collection devices 104, 108 can identifywhen the individual 102 and the caregiver 106 have convergent headorientation. In some examples, a straight line wireless signal, such asa Bluetooth signal, infrared signal, or RF signal, is passed between theindividual's wearable data collection device 104 and the caregiverwearable data collection device 108, such that a wireless receiveracknowledges when the two wearable data collection devices 104, 108 arepositioned in a substantially convergent trajectory.

The wearable data collection device 104, in some implementations, isconfigured to monitor physiological functions of the individual 102. Insome examples, the wearable data collection device 104 may collect heartand/or breathing rate data 116 e (or, optionally, electrocardiogram(EKG) data), electroencephalogram (EEG) data 116 f, and/orElectromyography (EMG) data 116 i). The wearable data collection device104 may interface with one or more peripheral devices, in someembodiments, to collect the physiological data. For example, thewearable data collection device 104 may have a wired or wirelessconnection with a separate heart rate monitor, EEG unit, or EMG unit. Inother embodiments, at least a portion of the physiological data iscollected via built-in monitoring systems. Unique methods fornon-invasive physiological monitoring are described in greater detail inrelation to FIG. 11. Optional onboard and peripheral sensor devices foruse in monitoring physiological data are described in relation to FIG.12.

In some implementations, during an evaluation session, the individual'swearable data collection device 104 gathers counts data 116 c related topatterns identified within other data 116. For example, the individual'sdata collection device 104 may count verbal (word and/or othervocalization) repetitions identified within the voice recording data 116a and movement repetitions identified in the head position data 116 dand/or the motion tracking data 116 h. The baseline analysis foridentifying repetitions (e.g., time span between repeated activity,threshold number of repetitions, etc.), in some embodiments, may betuned by educators and/or clinicians based upon baseline behavioranalysis of “normal” individuals or typical behaviors indicative ofindividuals with a particular clinical diagnosis such as ASD. Forexample, verbal repetition counts 116 c may be tuned to identifyrepetitive vocalizations separate from excited stuttering or otherrepetitive behaviors typical of children of an age or age range of theindividual. In another example, movement repetition counts 116 c maydistinguish from dancing and playful repetitive behaviors of a youngchild. Autism assessment, progress monitoring, and coaching all arecurrently done with little or no support via structured, quantitativedata which is one reason that rigorous counts 116 c are so veryimportant. Counts 116 c can include other types of behavior such asrocking, self-hugging, self-injurious behaviors, eye movements and blinkdynamics, unusually low-movement periods, unusually high-movementperiods, irregular breathing and gasping, behavioral or physiologicalsigns of seizures, irregular eating behaviors, and other repetitive orirregular behaviors.

In other implementations, rather than collecting the counts data 116 c,a remote analysis and data management system 118 (e.g., networkedserver, cloud-based processing system, etc.) analyzes a portion of thesession data 116 to identify at least a portion of the counts data 116 c(e.g., verbal repetition counts and/or movement repetition counts). Forexample, a session data analysis engine 120 of the remote analysis anddata management system 118 may analyze the voice recording data 116 a,motion tracking data 116 h, and/or head position data 116 d to identifythe verbal repetition counts and/or movement repetition counts.

In some implementations, the analysis is done at a later time. Forexample, the analysis and data management system 118 may archive thesession data 116 in an archive data store 122 for later analysis. Inother implementations, the session data and analysis engine 120 analyzesat least a portion of the session data 116 in real-time (e.g., throughbuffering the session data 116 in a buffer data store 124). For example,a real-time analysis of a portion of the session data 116 may besupplied to the evaluator 114 during the evaluation session. Thereal-time data analysis, for example, may be presented on the display112 as session information and statistics information 126. In someexamples, statistics information 126 includes presentation of raw datavalues, such as a graphical representation of heart rate or a graphicalpresentation of present EEG data. In other examples, statisticsinformation 126 includes data analysis output, such as a color-codedpresentation of relative excitability or stimulation of the subject(e.g., based upon analysis of a number of physiological factors) orgraphic indications of identified behaviors (e.g., an icon displayedeach time social eye contact is registered).

Session information and statistics information 126 can be used toperform behavioral decoding. Behavioral decoding is like languagetranslation except that it decodes the behaviors of an individual 102rather than verbal language utterances. For instance, a result of thesession data analysis 120 might be that a pattern emerges wherebyrepetitive vocalizations of a particular type as well as repeatedtouching the cheek are correlated, in the individual 102, with ambienttemperature readings below a certain temperature level, and thebehaviors cease when the temperature rises. Once this pattern has beenreliably measured by the system 100, upon future episodes of thosebehaviors, the system 100 could present to the caregiver 108 orevaluator 114 some information such as that the subject is likely toocold. The system 100 can also interface directly with control systems inthe environment, for instance in this case the system 100 may turn up athermostat to increase the ambient temperature. This example isillustrative of many possibilities for behavioral decoding. The system100 increases in ability to do behavioral decoding the longer itinteracts with the individual 102 to learn the behavioral language ofthe individual 102. Furthermore, the greater the total number ofindividuals interacting with the system 100, the greater the capacity ofthe system 100 to learn from normative data to identify stereotypicalcommunication strategies of individuals within subgroups of variousconditions, such as subgroups of the autism spectrum.

During an evaluation session, in an illustrative example, the caregiver106 is tasked with performing interactive tasks with the individual 102.Video recording data 116 j collected by the caregiver wearable datacollection device 108 is supplied to a computing system of the evaluator114 in real-time via the analysis and data management system 118 suchthat the evaluator 114 is able to see the individual 102 more or less“through the eyes of” the caregiver 108 during the evaluation session.The evaluator 114 may also receive voice recording data 116 a fromeither the caregiver wearable data collection device 108 or the subjectwearable data collection device 104.

Should the evaluator 114 wish to intercede during the evaluationsession, in some implementations, the evaluator 114 can call thecaregiver 106 using a telephone 128. For example, the caregiver 106 mayhave a cell phone or other personal phone for receiving telephonecommunications from the evaluator 114. In another example, the caregiverwearable computing device 108 may include a cellular communicationssystem such that a telephone call placed by the evaluator 114 isconnected to the caregiver wearable computing device 108. In thismanner, for example, the caregiver 108 may receive communications fromthe evaluator 114 without disrupting the evaluation session.

In other implementations, a computer-aided (e.g., voice over IP, etc.)communication session is established between the evaluator 114 computingsystem and the caregiver wearable data collection device 108. Forexample, the analysis and data management system 118 may establish andcoordinate a communication session between the evaluator system and thecaregiver wearable data collection device 108 for the duration of theevaluation system. Further, the analysis and data management system 118,in some embodiments, may collect and store voice recording data ofcommentary supplied by the evaluator 114.

In some examples, the evaluator 114 may communicate with the caregiver106 to instruct the caregiver 106 to perform certain interactions withthe individual 102 or to repeat certain interactions with the individual102. Prior to or at the end of an evaluation session, furthermore, theevaluator 114 may discuss the evaluation with the caregiver 106. In thismanner, the caregiver 106 may receive immediate feedback and support ofthe evaluator 114 from the comfort of her own home.

FIG. 1B is a block diagram of an example system 150 for evaluation andtraining of the individual 102 using the wearable data collection device104. Data 116 collected by the wearable data collection device 104 (and,optionally or alternatively, data collected by the caregiver datacollection device 108 described in relation to FIG. 1A) is used by anumber of algorithms 154 developed to analyze the data 116 and determinefeedback 156 to provide to the individual 102 (e.g., via the wearabledata collection device 104 or another computing device). Furthermore,additional algorithms 532, 534, 536, 538, 540, 542, and 544 described inrelation to FIG. 5B and/or algorithms 910 and 912 described in relationto FIG. 9 may take advantage of components of the system 150 inexecution. The algorithms 154 may further generate analysis information158 to supply, along with at least a portion of the data 116, tolearning engines 162. The analysis information 158 and data 116, alongwith learning information 164 generated by the learning engines 162, maybe archived as archive data 122 for future use, such as for pooledstatistical learning. The learning engines 162, furthermore, may providelearned data 166 and, potentially, other system updates for use by thewearable data collection device 104. The learned data 166, for example,may be used by one or more of the algorithms 154 residing upon thewearable data collection device 104. A portion or all of the dataanalysis and feedback system 152, for example, may execute upon thewearable data collection device 104. Conversely, in someimplementations, a portion or all of the data analysis and feedbacksystem 152 is external to the wearable data collection device 104. Forexample, certain algorithms 154 may reside upon a computing device incommunication with the wearable data collection device 104, such as asmart phone, smart watch, tablet computer, or other personal computingdevice in the vicinity of the individual 102 (e.g., belonging to acaregiver, owned by the individual 102, etc.). Certain algorithms 154,in another example, may reside upon a computing system accessible to thewearable data collection device 104 via a network connection, such as acloud-based processing system.

The algorithms 154 represent a sampling of potential algorithmsavailable to the wearable data collection device 104 (and/or thecaregiver wearable data collection device 108 as described in relationto FIG. 1A). The algorithms 154 include an audio recording analysisalgorithm 154 a, a video recording analysis algorithm 154 b, an eyemotion analysis algorithm 154 c, a head motion analysis algorithm 154 d,a social eye contact identifying algorithm 154 e, a feedbackpresentation algorithm 154 f, a subject response analysis algorithm 154g, a vocalized repetition tracking algorithm 154 h (e.g., to generate aportion of the counts data 116 c illustrated in FIG. 1A), a movementrepetition tracking algorithm 154 i (e.g., to generate a portion of thecounts data 116 c illustrated in FIG. 1A), an object identificationalgorithm 154 j, a physiological state analysis algorithm 154 k, anemotional state analysis algorithm 154 l, a social response validationalgorithm 154 m, a desired response identification algorithm 154 n, asocial event identification algorithm 154 o, and a verbal responsevalidation engine 154 p. Versions of one or more of the algorithms 154may vary based upon whether they are executed upon the individual'swearable data collection device 104 or the caregiver wearable datacollection device 108. For example, the social eye contactidentification algorithm 154 e may differ when interpreting videorecording data 116 b supplied from the viewpoint of the individual 102as compared to video recording data 116 b supplied from the viewpoint ofthe caregiver 106 (illustrated in FIG. 1A).

The algorithms 154 represent various algorithms used in performingvarious methods described herein. For example, method 600 regardingidentifying objects labeled with standardized index elements (describedin relation to FIG. 6A) and/or method 610 regarding extractinginformation from objects with standardized index elements (described inrelation to FIG. 6B), may be performed by the object identificationalgorithm 154 j. Step 662 of method 630 (described in relation to FIG.6D) regarding validating the subject's response may be performed by theverbal response validation algorithm 154 p. Step 664 of method 630(described in relation to FIG. 6D) regarding providing feedbackregarding the subject's response may be performed by the feedbackpresentation algorithm 154 f. Step 704 of method 700 regarding detectionof a socially relevant event, described in relation to FIG. 7A, may beperformed by the social event identification algorithm 154 o. Step 716of method 700 regarding determination of a desired response to asocially relevant event may be performed by the desired responseidentification algorithm 154 n. Step 718 of method 700 regardingcomparison of the subject's actual response may be performed by thesocial response validation algorithm 154 m. Step 740 of method 700regarding reviewing physiological data, described in relation to FIG.7B, may be performed by the physiological state analysis algorithm 154k. Step 802 of method 800 regarding identification of faces in videodata, described in relation to FIG. 8, may be performed by the videorecording analysis algorithm 154 b. Step 810 of method 800 regardingidentification of social eye contact may be performed by the social eyecontact identification algorithm 154 e. The social eye contactidentification algorithm 154 e, in turn, may utilize the eye motionanalysis engine 154 c and/or the head motion analysis engine 154 d inidentifying instances of social eye contact between the individual 102and another individual. Step 816 of method 800 regarding ascertaining anindividual's reaction to feedback may be performed by the subjectresponse analysis algorithm 154 g. Step 1006 of method 1000 regardingidentifying an emotional state of an individual, described in relationto FIG. 10A, may be performed by the emotional state analysis algorithm154 l. Step 1010 of method 1000 regarding analyzing audio data foremotional cues may be performed by the audio recording analysisalgorithm 154 a.

The algorithms 154, in some implementations, are utilized by varioussoftware modules 302 described in relation to FIG. 3A. For example, asocial eye contact training module 302 a may utilize the social eyecontact identification algorithm 154 e. A socially relevant eventtraining module 302 b, in another example, may utilize the socialresponse validation algorithm 154 m, the desired response identificationalgorithm 154 n, and/or the social event identification algorithm 154 o.

The algorithms 154, in some implementations generate analysisinformation 158 such as, for example, the derived session data 306illustrated in FIG. 3A. The analysis information 158 may be provided inreal time and/or in batch mode to a learning and statistical analysissystem 160 including the learning engines 162. The learning engines 162,for example, may include the statistical analysis software modules 352illustrated in FIG. 3B. A portion of the statistical analysis system 160may execute upon the wearable data collection device 104. Conversely, insome implementations, a portion or all of the statistical analysissystem 160 is external to the wearable data collection device 104. Forexample, certain learning engines 162 may reside upon a computing devicein communication with the wearable data collection device 104, such as asmart phone, smart watch, tablet computer, or other personal computingdevice in the vicinity of the individual 102 (e.g., belonging to acaregiver, owned by the individual 102, etc.). The statistical analysissystem 160, in another example, may reside upon a computing systemaccessible to the wearable data collection device 104 via a networkconnection, such as a cloud-based processing system.

The learning engines 162, in some implementations, generate learninginformation 164. For example, as illustrated in FIG. 3B, statisticallylearned data 356 may include social interaction patterns 356 e. Thelearning engines 162 may execute a subject social interaction progresssoftware module 352 a to track progress of interactions of theindividual 102 with the caregiver 106. Further, statistically learneddata 356, in some implementations, may lead to system updates 166presented to improve and refine the performance of the wearable datacollection device 104. Statistically learned data 356, in someimplementations, can be used to predict acting out or episodes in peoplewith ASD. In some implementations, statistically learned data 356 can beused to predict, based on current conditions and environmental featuresas well as physiological or behavioral signals from the subject,unwellness or health episodes such as seizures or migraine onset orheart attacks or other cardiovascular episodes, or other outcomes suchas are related to ASD. Statistically learned data 356 can be used toprovide behavioral decoding. For instance, statistically learned data356 may indicate that one type of self-hitting behavior plus a specificvocalization occurs in an individual 102 most frequently before mealtimes, and these behaviors are most pronounced if a meal is delayedrelative to a regular meal time, and that they are extinguished as soonas a meal is provided and prevented if snacks are given before a regularmeal. In this context, these behaviors may be statistically associatedwith hunger. The prior example is simplistic in nature—a benefit ofcomputer-based statistical learning is that the statistical learningdata 356 can allow the system to recognize patterns that are lessobvious than this illustrative example. In the present example, atfuture times, statistical learning data 356 that resulted in recognitionof a pattern such as mentioned can provide for behavioral decoding suchas recognizing the behaviors as an indicator that the individual 102 islikely hungry.

Behavioral decoding can be used for feedback and/or for intervention.For instance, in terms of feedback, the system, in some implementations,provides visual, textual, auditory or other feedback to the individual102, caregiver 106, and/or evaluator 114 (e.g., feedback identifyingthat the individual 102 is likely hungry). Behavioral decoding can alsobe used for intervention. For instance, in this case, when theaforementioned behaviors start emerging, a control signal can be sentfrom the system 100 to trigger in intervention that will reduce hunger,such as in this case ordering of food or instruction to the caregiver toprovide food.

Turning to FIGS. 2A and 2B, a swim lane diagram illustrates a method 200for conducting an evaluation session through a caregiver system 204 anda user system 202 monitored by an evaluator system 208. Informationpassed between the evaluator system 208 and either the caregiver system204 or the user system 202 is managed by an analysis system 206. Thecaregiver system 204 and/or the user system 202 include a wearable datacollection device, such as the wearable data collection devices 104 and108 described in relation to FIG. 1A. The evaluation system 208 includesa computing system and display for presentation of information collectedby the wearable data collection device(s) to an evaluator, such as theevaluator 114 described in relation to FIG. 1A. The analysis system 206includes a data archival system such as the data buffer 128 and/or thedata archive 122 described in relation to FIG. 1A, as well as ananalysis module, such as the session data analysis engine 120 describedin relation to FIG. 1A.

In some implementations, the method 200 begins with initiating anevaluation session (210) between the caregiver system 204 and the usersystem 202. An evaluator may have defined parameters regarding theevaluation session, such as a length of time, activities to includewithin the evaluation session, and props or objects to engage withduring the evaluation session. In initiating the evaluation session, asoftware application functioning on the caregiver system 204 maycommunicate with a software application on the user system 202 tocoordinate timing and initialize any data sharing parameters for theevaluation session. In a particular example, the caregiver system 204may issue a remote control “trigger” to the user system 202 (e.g.,wearable data collection device) to initiate data collection by the usersystem 202. Meanwhile, the caregiver system 204 may initiate datacollection locally (e.g., audio and/or video recording).

In some implementations, initiating the evaluation session furtherincludes opening a real-time communication channel with the evaluatorsystem 208. For example, the real-time evaluation session may be openbetween the caregiver system 204 and the evaluator system 208 and/or theuser system 202 and the evaluator system 208. In some implementations,the caregiver system 204 initiates the evaluation session based upon aninitiation trigger supplied by the evaluator system 208.

In some implementations, session data is uploaded (212) from the usersystem 202 to the analysis system 206. For example, data collected byone or more modules functioning upon the user system 202, such as avideo collection module and an audio collection module, may be passedfrom the subject system 202 to the analysis system 206. The data, insome embodiments, is streamed in real-time. In other embodiments, thedata is supplied at set intervals, such as, in some examples, after athreshold quantity of data has been collected, after a particular phaseof the session has been completed, or upon pausing an ongoing evaluationsession. The data, in further examples, can include eye tracking data,motion tracking data, EMG data, EEG data, heart rate data, breathingrate data, and data regarding subject repetitions (e.g., repetitivemotions and/or vocalizations).

Furthermore, in some implementations, session data is uploaded (214)from the caregiver system 204 to the analysis system 206. For example,audio data and/or video data collected by a wearable data collectiondevice worn by the caregiver may be uploaded to the analysis system 206.Similar to the upload from the subject system 202 and the analysissystem 206, data upload from the caregiver system 204 to the analysissystem 206 may be done in real time, periodically, or based upon one ormore triggering events.

In some implementations, the analysis system 206 analyzes (216) thesession data. Data analysis can include, in some examples, identifyinginstances of social eye contact between the individual and thecaregiver, identifying emotional words, and identifying vocalization ofthe subject's name. The analysis system 206, in some embodiments,determines counts of movement repetitions and/or verbal repetitionsduring recording of the individual's behavior. Further, in someembodiments, data analysis includes deriving emotional state of theindividual from one or more behavioral and/or physiological cues (e.g.,verbal, body language, EEG, EMG, heart rate, breathing rate, etc.). Forexample, the analysis system 206 may analyze the reaction and/oremotional state of the individual to the vocalization of her name. Theanalysis system 206, in some embodiments, further analyzes caregiverreactions to identified behaviors of the individual such as, in someexamples, social eye contact, repetitive behaviors, and vocalizations.For example, the analysis system 206 may analyze body language,emotional words, and/or vocalization tone derived from audio and/orvideo data to determine caregiver response.

In some implementations, analyzing the session data (216) includesformatting session data into presentation data for the evaluator system208. For example, the analysis system 206 may process heart rate datareceived from the user system 202 to identify and color code instancesof elevated heart rate, as well as preparing presentation of the heartrate data in graphic format for presentation to the evaluator. Ifprepare in real time, the session data supplied by the user system 202and/or the caregiver system 204 may be time delayed such that rawsession information (e.g., video feed) may be presented to the evaluatorsimultaneously with processed data feed (e.g., heart rate graph).

The analysis system 206, in some implementations, archives at least aportion of the session data. For example, the session data may bearchived for review by an evaluator at a later time. In another example,archived system data may be analyzed in relation to session data derivedfrom a number of additional subjects to derived learned statistical data(described in greater detail in relation to FIG. 3B).

In some implementations, the analysis system 206 provides (218) sessioninformation, including raw session data and/or processed session data,to the evaluator system 208. At least a portion of the session datacollected from the user system 202 and/or the caregiver system 204, inone example, is supplied in real time or near-real time to the evaluatorsystem 208. As described above, the session information may includeenhanced processed session data prepared for graphical presentation tothe evaluator. In another example, the evaluator system 208 may requestthe session information from the analysis system 206 at a later time.For example, the evaluator may review the session after the individualand caregiver have completed and authorized upload of the session to theanalysis system. In this manner, the evaluator may review session dataat leisure without needing to coordinate scheduling with the caregiver.

In some implementations, if the evaluator is reviewing the sessioninformation in near-real-time, the evaluator system 208 issues (222) aninstruction to the caregiver system 204. The evaluator, for example, mayprovide verbal instructions via a telephone call to the caregiver system204 or an audio communication session between the evaluator system 208and the caregiver system 204. For example, a voice data session may beestablished between the evaluator system 208 and the caregiver'swearable data collection device. In another example, the evaluatorsystem 208 may supply written instructions or a graphic cue to thecaregiver system 204. In a particular example, a graphic cue may bepresented upon a heads-up display of the caregiver's wearable datacollection device to prompt the caregiver to interact with theindividual using a particular object.

Rather than issuing an instruction, in some implementations theevaluator system 208 takes partial control of either the caregiversystem 204 or the user system 202. In some examples, the evaluatorsystem 208 may assert control to speak through the user system 202 tothe individual or to adjust present settings of the wearable datacollection device of the caregiver. In taking partial control of thecaregiver system 204 or the user system 202, the evaluator system 208may communicate directly with either the caregiver system 204 or theuser system 202 rather than via the relay of the analysis system 206.

Similarly, although the instruction, as illustrated, bypasses theanalysis system 206, the communication session between the evaluatorsystem 208 and the caregiver system 204, in some implementations, isestablished by the analysis system 206. The analysis system 206, in someembodiments, may collect and archive a copy of any communicationssupplied to the caregiver system 204 by the evaluator system 208.

In some implementations, the caregiver system 204 performs (224) theinstruction. For example, the instruction may initiate collection ofadditional data and/or real-time supply of additional data from one ofthe caregiver system 204 and the subject system 202 to the evaluatorsystem 208 (e.g., via the analysis system 206). The evaluator system208, in another example, may cue a next phase on the evaluation sessionby presenting instructional information to the caregiver via thecaregiver system 204. For example, upon cue by the evaluator system 208,the caregiver system 204 may access and present instructions forperforming the next phase of the evaluation session by presentinggraphical and/or audio information to the caregiver via the wearabledata collection device.

In some implementations, the user system 202 uploads (226) additionalsession data and the caregiver system 204 uploads (228) additionalsession data. The data upload process may continue throughout theevaluation session, as described, for example, in relation to steps 212and steps 214.

Turning to FIG. 2B, in some implementations, the evaluator enters (230)evaluation data via the evaluator system 208. For example, the evaluatormay include comments, characterizations, caregiver feedback, and/orrecommendations regarding the session information reviewed by theevaluator via the evaluator system 208.

In some implementations, the evaluator system 208 provides (232) theevaluation data to the analysis system 206. The evaluation data, forexample, may be archived along with the session data. At least a portionof the evaluation data, furthermore, may be supplied from the analysissystem 206 to the caregiver system 204, for example as immediatefeedback to the caregiver. In some embodiments, a portion of theevaluation data includes standardized criteria, such that the sessiondata may be compared to session data of other individuals characterizedin a same or similar manner during evaluation.

In some implementations, the analysis system 206 archives (234) thesession and evaluation data. For example, the session and evaluationdata may be uploaded to long term storage in a server farm or cloudstorage area. Archival of the session data and evaluation data, forexample, allows data availability for further review and/or analysis.The session data and evaluation data may be anonymized, secured, orotherwise protected from misuse prior to archival.

In some implementations, the analysis system 206 statistically analyzes(236) the archived data from multiple sessions. In one example, archivedsession data may be compared to subsequent session data to reinforcecharacterizations or to track progress of the individual. In anotherexample, as described above, the session data may be evaluated inrelation to session data obtained from further individuals to derivelearning statistics regarding similarly characterized individuals. Theevaluation data supplied by the evaluator in step 230, in one example,may include an indication of desired analysis of the session data. Forexample, the session data may be compared to session data collectedduring evaluation of a sibling of the subject on a prior occasion.

In some implementations, the analysis system 206 provides (238) analysisinformation derived from the archived session data to the evaluatorsystem 208. For example, upon analyzing the session data in view ofprior session data with the same individual, progress data may besupplied to the evaluator system 208 for review by the evaluator.

FIG. 3A is a block diagram of a computing system 300 for training andfeedback software modules 302 for execution in relation to a wearabledata collection device. The training and feedback software modules 302incorporate various raw session data 304 obtained by a wearable datacollection device, and generate various derived session data 306. Thetraining and feedback software modules 302, for example, may includesoftware modules capable of executing on any one of the subject wearabledata collection device 104, the caregiver wearable data collectiondevice 108, and the analysis and data management system 118 of FIG. 1A.Further, at least a portion of the training and feedback softwaremodules 302 may be employed in a system 500 of FIG. 5A, for example in awearable data collection device 504 and/or a learning data analysissystem 520, or in a system 1100 of the FIG. 11, for example in awearable data collection device 1104 and/or a learning data analysissystem 1118. The raw session data 304, for example, may represent thetype of session data shared between the subject system 202 or thecaregiver system 204 and the analysis system 206, as described inrelation to FIG. 2A.

FIG. 3B is a block diagram of a computing system 350 for analyzing andstatistically learning from data collected through wearable datacollection devices. The archived session data 354 may include datastored as archive data 122 as described in FIG. 1A and/or data stored asarchive data 1122 as described in FIG. 11. For example, the analysissystem 206 of FIG. 2B, when statistically analyzing the archived data instep 236, may perform one or more of the statistical analysis softwaremodules 352 upon a portion of the archived session data 354.

FIG. 4 is a flow chart of an example method 400 for conducting anevaluation session using a wearable data collection device donned by acaregiver of an individual being evaluated for Autism Spectrum Disorder.The method 400, for example, may be performed independent of anevaluator in the comfort of the caregiver's home. The caregiver may besupplied with a kit including a wearable data collection device andinstructions for performing an evaluation session. The kit mayoptionally include a wearable data collection device for the individual.

In some implementations, the method 400 begins with the caregiverdonning the wearable data collection device (402). Examples of awearable data collection device are described in relation to FIG. 1A.The wearable data collection device, for example, may include ahead-mounted lens for a video recording system, a microphone for audiorecording, and a head-mounted display. Further, the wearable datacollection device may include a storage medium for storing datacollected during the evaluation session.

In some implementations, the evaluation session is initiated (404). Uponpowering and donning the wearable data collection device, or launchingan evaluation session application, the evaluation session may beinitiated. Initiation of the evaluation session may include, in someembodiments, establishment of a communication channel between thewearable data communication device and a remote computing system.

In some implementations, instructions are presented for a first phase ofevaluation (406). The instructions may be in textual, video, and/oraudio format. Instructions, for example, may be presented upon aheads-up display of the wearable data collection device. If acommunication channel was established with the remote computing system,the instructions may be relayed to the wearable data communicationdevice from the remote computing system. In other embodiments, theinstructions may be programmed into the wearable data communicationdevice. The evaluation kit, for example, may be preprogrammed to directthe caregiver through an evaluation session tailored for a particularindividual (e.g., first evaluation of a 3-year-old male lacking verbalcommunication skills versus follow-on evaluation of a 8-year-old femaleperforming academically at grade level). In another example, thecaregiver may be prompted for information related to the individual, anda session style may be selected based upon demographic and developmentalinformation provided. In other implementations, rather than presentinginstructions, the caregiver may be prompted to review a booklet orseparate video to familiarize himself with the instructions.

The evaluation session, in some implementations, is performed as aseries of stages. Each stage for example, may include one or moreactivities geared towards encouraging interaction between the caregiverand the individual. After reviewing the instructions, the caregiver maybe prompted to initiate the first phase of evaluation. If the phase isinitiated, in some implementations, audio and video recording of theevaluation phase is initiated (410). The wearable data collectiondevice, for example, may proceed to collect data related to theidentified session.

In some implementations, upon conclusion of the phase, the caregiver isprompted for approval (412). The caregiver may be provided theopportunity to approve the phase of evaluation, for example, based uponwhether the phase was successfully completed. A phase may have failed tocomplete successfully, in some examples, due to unpredicted interruption(e.g., visitor arriving at the home, child running from the room andrefusing to participate, etc.).

In some implementations, if the phase has not been approved (414), thephase may be repeated by re-initiating the current phase (408) andrepeating collection of audio and video recording (410). In this manner,if the evaluation session phase is interrupted or otherwise failed torun to completion, the caregiver may re-try a particular evaluationphase.

Upon approval by the caregiver of the present phase (414), in someimplementations, session data associated with the particular phase isstored and/or uploaded (416). The data, for example, may be maintainedin a local storage medium by the wearable data collection device oruploaded to the remote computing system. Metadata, such as a sessionidentifier, phase identifier, subject identifier, and timestamp, may beassociated with the collected data. In some implementations, for storageor transfer, the wearable data collection device secures the data usingone or more security algorithms to protect the data from unauthorizedreview.

In some implementations, if additional phases of the session exist(418), instructions for a next phase of the evaluation are presented(406). As described above in relation to step 406, for example, thewearable data collection device may present instructions for caregiverreview or prompt the caregiver to review separate instructions relatedto the next phase.

In some implementations, at the end of each phase, the caregiver may beprovided the opportunity to suspend a session, for example to allow theindividual to take a break or to tend to some other activity prior tocontinuing the evaluation session. In other implementations, thecaregiver is encouraged to proceed with the evaluation session, forexample to allow an evaluator later to review the individual's responsesas phase activities are compounded.

If no additional phases exist in the evaluation session (418), in someimplementations, remaining session data is uploaded or stored (420) asdescribed in step 416. If the phase data was previously stored locallyon the wearable data collection device, at this point, the entiresession data may be uploaded to the remote computing system. In otherembodiments, the session data remains stored on the wearable datacollection device, and the wearable data collection device may bereturned for evaluation and reuse purposes. In addition to the sessiondata, the caregiver may be prompted to provide additional data regardingthe session, such as a session feedback survey or comments regarding theindividual's participation in the evaluation session compared to theindividual's typical at-home behaviors. This information may be uploadedor stored along with the data collected for each evaluation phase.

FIG. 5A is a block diagram of an example environment 500 for augmentedreality learning, coaching, and assessment using a wearable datacollection device 504. As illustrated, the wearable data collectiondevice 504 shares many of the same data collection features 116 as thewearable data collection devices 104 and 108 described in relation toFIG. 1A. Additionally, the wearable data collection device includes datacollection and interpretation features 506 configured generally foridentifying objects and individuals within a vicinity of an individual502 and for prompting, coaching, or assessing interactions between theindividual 502 and those objects and individuals within the vicinity.

In some implementations, the example environment includes a remoteanalysis system 514 for analyzing the data 116 and/or 506 using one ormore learning data analysis modules 520 executing upon a processingsystem 518 (e.g., one or more computing devices or other processingcircuitry). The learning data analysis module(s) 520 may store rawand/or analyzed data 116, 506 as session data 516 in a data store 524.Further, the remote analysis system 514 may archive collected data 116and/or 506 in a data archive 522 for later analysis or for crowd-sourcedsharing to support learning engines to enhance performance of thelearning data analysis modules 520.

In addition to or in replacement of the learning data analysis module(s)520, in some implementations, the processing system 518 includes one ormore language and communication algorithms 530 (e.g., software,firmware, and/or hardware-based computing algorithms designed to assess,train, and coach the individual 502 in language and communicationskills), illustrated in FIG. 5B. Rather than residing in the remoteanalysis system 514, in some implementations, one or more of thealgorithms 530 (or feature portions thereof) are executed upon thewearable data collection device and/or on a peripheral computing devicein communication with the wearable data collection device.

Turning to FIG. 5B, the language and communication algorithms 530include a set of reading tools 532, a set of speech-filtering tools 534,a set of conversational tools 536, a set of communicative gesture tools538, a set of speech coaching tools 540, a set of interpersonalcommunication tools 542, and a teleprompter algorithm 544. Although eachset of tools 532-542 includes individual topic algorithms, in otherimplementations, one or more of the algorithms 532-542 may be combined.Additionally, a particular algorithm 532-544 may be divided into two ormore algorithm modules. The algorithms 532-544, together, provide alanguage tool set configured to support reading, linguistics,interpersonal communications, and speech understanding.

Beginning with the reading tools 532, a machine vision language tutoralgorithm 532 a, in some implementations, supports recognition andlearning modules incorporating machine-encoded objects within thevicinity of the individual 502. Turning to FIG. 5A, the machine visionlanguage tutor algorithm 532 a may include, for example, the ability toidentify encoded objects within the vicinity of the wearable datacollection device 504. For example, the machine vision language tutoralgorithm 532 a may scan the immediate vicinity of the individual 502wearing the wearable data collection device 504 to identify objectsencoded with standardized index elements 512, such as, in some examples,a two-dimensional barcode, three-dimensional barcode, QR code,radio-frequency identification (RFID) tags, and other machine-readablelabels or electronically transmitting smart labels. As illustrated, aball object 508 includes an RFID tag element 512 a and a clock object510 includes a QR code element 512 b. Each standardized index element512, in turn, may be encoded with or otherwise identify a unique objectindex 506 a. In one example, the machine vision language tutor algorithm532 a, executing upon the wearable data collection device 504 or acomputing device in communication with the wearable data collectiondevice (e.g., the processing system 518 or a local computing device suchas a smart phone, tablet computer, etc.) 504 may use one or morehardware, firmware, or software elements of the wearable data collectiondevice to scan the immediate vicinity to collect object indices 506 aassociated with each encoded object 508, 510. In a particular example,the machine vision language tutor algorithm 532 a may use an RFIDscanner feature of the wearable data collection device 504 to scan thevicinity to identify the RFID tag 512 a. In another example, the machinevision language tutor algorithm 532 a may analyze video recording data116 b captured by the wearable data collection device 504 or a computingsystem in communication with the wearable data collection device 504 toidentify the standardized index elements 512 (e.g., QR codes or barcodes). In other examples, the machine vision language tutor algorithm532 a uses machine-vision processes, machine-hearing, or other signalprocessing abilities of the wearable data collection device 504 toidentify objects with standardized index elements in the vicinity. Toimprove recognition of objects encoded with standardized index elementswithin the vicinity, in some embodiments, the machine vision languagetutor algorithm 532 a may use two or more separate methods ofidentifying items. The machine vision language tutor algorithm 532 a maycross-reference the objects identified using a first recognition method,for example, with the objects identified using a second recognitionmethod.

In some implementations, each standardized index element 512 is embeddedwith a particular identifier (e.g., substring) that is otherwiseunlikely to occur in that particular type of index element, such thatthe identifier can be used to identify standardized index elementscreated for use with the wearable data collection device 504. Forexample, while scanning the vicinity for standardized index elements,the machine vision language tutor algorithm 532 a can ignore thoselabels (e.g., QR codes, RFID tags) lacking the identifier.

In some implementations, the machine vision language tutor algorithm 532a matches object data 506 f to each object index 506 a. For example, themachine vision language tutor algorithm 532 a may apply the object index506 a to a look-up table to derive associated object data 506 fregarding the encoded object. In the event that the object data 506 faccessed depends upon a particular functional mode of the machine visionlanguage tutor algorithm 532 a and/or the wearable data collectiondevice 504, the machine vision language tutor algorithm 532 a may accessa mode-specific look-up table to derive associated object data 506 f. Inanother example, the machine vision language tutor algorithm 532 a mayaccess a database to derive multiple representations of a particulardata group, for example object data 506 f including terms for an item ina number of foreign languages. In another example, a smart label such asan RFID tag may include embedded object data 506 f which can be read bythe machine vision language tutor algorithm 532 a.

The machine vision language tutor algorithm 532 a, in someimplementations, presents a portion of the derived object data 506 f tothe individual 502. For example, video augmentation data 506 b may beused by a video augmentation module of the machine vision language tutoralgorithm 532 a to portray the names of each object in a display regionof the wearable data collection device 504 as written words floatingabove or upon each object. In another example, the machine visionlanguage tutor algorithm 532 a may cause the names of each object may beintoned audibly to the individual 502, for example through a soundsystem of the wearable data collection device 504 that includes aheadphone or bone-conduction speaker. In further examples, the machinevision language tutor algorithm 532 a may present derived object data506 f associated with the object to the individual 502, such as atick-tock and/or chiming sound associated with a clock.

In some implementations, prior to presenting any object data 506 frelated to the acquired object indices 506 a, the individual 502 mayfirst select a desired object. Selection, in some examples, may beaccomplished via a hand gesture, head gesture, eye movement (e.g.,double blink), audible command, thought pattern, or other instructionissued by the individual 502 via an input system of the wearable datacollection device 504. Upon selection of one of the objects 508, 510,for example, the video augmentation module of the machine visionlanguage tutor algorithm 532 a may present the individual 502 with anaugmented video representation of the field of vision, including objectdata 506 f regarding the selected object 508. In another example, anaudio feedback module of the machine vision language tutor algorithm 532a may play audible object data 506 f regarding the selected object 508,510.

In some implementations, selection of an object triggers a deepinformation retrieval module of the machine vision language tutoralgorithm 532 a. For example, in the context of a chemistry lab, initialobject data 506 f may include the name of a chemical compound, while asecond (deeper) level of object data 506 f may include a chemistryinformation sheet regarding the specific compound. Rather thanpresenting the deeper level object data 506 f via the wearable datacollection device 504, in some embodiments the machine vision languagetutor algorithm 532 a may redirect the deeper level object data 506 f toa separate computing device, such as, in some examples, a smart phone,tablet computer, laptop computer, or smart television. The wearable datacollection device 504, in some embodiments, shares the object data 506 fwith the separate computing device through a wireless communicationslink, such as a Wi-Fi or Bluetooth connection.

The type and style of presentation of object data 506 f, in someimplementations, depends upon a mode of operation of the wearable datacollection device 504 or the machine vision language tutor algorithm 532a, potentially involving one or more additional software modules oralgorithms currently active upon the wearable data collection device504. The mode may in part represent a level of complexity of vocabulary,such as a grade level or reading achievement level. Other modegranulations, in some examples, may include picture presentation versusword presentation, parts of speech, category labels for the objects(which can be partially overlapping) such as animal-word or long-word orconcrete-word or happy-word or any other semantic or syntactic orpragmatic category, sentence fragments incorporating informationregarding the objects, sentences with words for the objects in them,auditory representations of the objects (e.g., tick-tock for the clockobject 510), visual representations of the type of object or category ofobject, olfactory representations of objects (e.g., flowers, foods,etc.), tactile representations of the objects, haptic representations ofthe objects, or any mix of types of object representations. In someembodiments, object representations can include items that relate to butmight not fully represent the particular object. In one example, uponselection of a particular object 508, 510, the machine vision languagetutor algorithm 532 a may present the individual 502 with a foreignlanguage lesson incorporating the selected object 508 or 510, such asthe Spanish word for ball or a sentence describing the present time ofday in Mandarin Chinese. The foreign language lesson, in some examples,may involve execution of a single word reading algorithm 532 b and/or agraphic enhanced vocabulary algorithm 532 d, described in greater detailin relation to FIG. 5B.

In some implementations, a caregiver, teacher, or other user associateseach label with particular object data. For example, a user may printlabels to apply to objects around the home, associating each object withat least a first piece of data (e.g., printed name or vocalized name).In another example, the user or caregiver may purchase labels (e.g.,sheets of sticker labels), scan each label with a standardized indexelement scanning application (e.g., built into the wearable datacollection device or downloadable to a personal computing deviceincluding scanning capability such as a smart phone), and associate eachscanned label with object data. The user or caregiver may then apply thelabels to the associated objects. In this manner, a user or caregivermay customize information gathering within a chosen vicinity (e.g.,classroom, child's bedroom, clinical office, etc.).

The mode of operation may further involve receiving responses from theindividual 502 regarding presented object data 506 f. For example, asillustrated, the word “clock” 526 is intoned to the individual 502. Thecurrently active software module may be a verbal skill building module(e.g., English language or foreign language mode) anticipatingrepetition of the intoned word. Upon identifying a spoken responsewithin voice recording data 116 a, the verbal skill building module mayvalidate the response and store the result (e.g., proximity inpronunciation) as response validation data 506 c. Furthermore, theverbal skill building module may present feedback data 506 e to theindividual 502 regarding relative success of pronunciation. The feedbackdata 506 e, in some examples, can include a visual indication (e.g.,green check or red “X” presented in a heads up display) and/or audibleindication (e.g., fanfare or buzzer). If the software module ispresenting a language lesson game, in some implementations, progresstracking data 506 d is collected to track the success of the individual502 in learning verbalizations associated with the labeled objects 508,510. A single word reading algorithm 532 b, in another example, maybehave similarly to the series of events described above in relation tothe verbal skill building module 536 c, but presenting a graphicillustration of the word “clock” 526 in lieu of the intonation.

In some implementations, interactions of the individual 502 with labeledobjects 508, 510 can take place in the form of a game. For example,video augmentation data 506 b may include an augmentation style toconvert the vicinity to a virtual reality zone having a particularpresentation style. The presentation style, in some examples, caninclude a line-drawn version of the vicinity, a cartoon-drawn version ofthe vicinity, or a simplified version of the vicinity, for example wherethe majority of the scene is reduced to wire frame with only the objects508 and 510 presented in full color. In another example, thepresentation style may include a full color version of the videorecording data 116 b with augmentation of the objects 508, 510 (e.g.,cartoon drawing, outlined in colorful lines, sparkling, jiggling, etc.).

In some implementations, the machine vision language tutor algorithm 532a, executing upon or in conjunction with the wearable data collectiondevice 504, correlates identified object indices 506 a with the locationcoordinates 506 g of the index elements 512 at the time of acquisition.The location coordinates 506 g, for example, may include two-dimensionalcoordinates (e.g., within a video frame reference) or three-dimensionalcoordinates (e.g., with respect to the individual 102). Identificationof the object indices 506 a, furthermore, may be associated with atime-date stamp identifying the time of acquisition. The locationcoordinates can be factored into presenting information to theindividual 502 related to the objects 508, 510. For example, if the ballobject 508 had been moving when the wearable data collection device 504registered the index element 512 a, the machine vision language tutoralgorithm 532 a could present a representation of the ball object 508 tothe individual 502 showing the ball 508 in a different location based onthe passage of time and motion characteristics of the ball 508 (e.g., asidentified within the video recording data 116 b). Likewise, the machinevision language tutor algorithm 532 a may identify movement of the headof the individual 502 based upon sensor elements within and/orcoordinating with the wearable data collection device 504 (e.g., viamotion tracking data 116 h and/or head position data 116 d) between thetime of acquisition of the index element 512 a and time of output ofobject data 506 f regarding the ball object 508 to the individual 502.Based upon the identified movements, the machine vision language tutoralgorithm 532 a may adjust the object data 506 f accordingly. Forinstance in the case of a visual image, the machine vision languagetutor algorithm 532 a can cause a shift in the visual image to representthe current head gaze direction as opposed to the one at the time ofacquisition—a form of motion correction.

Head gaze direction 116 d and subject motion data 116 h, in someimplementations, may be used by the machine vision language tutoralgorithm 532 a to identify which object data 506 f to present to theindividual 502. For example, based upon a present gaze trajectory of theindividual 502 (e.g., based upon head position data 116 d and/or eyetracking data 116 g), object data 506 f regarding the clock object 510,rather than object data 506 f regarding the ball object 508, may bepresented to the individual 502.

In some implementations, the machine vision language tutor algorithm 532a uses the location coordinates 506 g of the index elements 512 toidentify three-dimensional locations of the objects 508, 510 withreference to the individual 502. For example, location coordinates 506 gmay be derived from triangulation of video recording data 116 b obtainedat multiple angles. In another example, location coordinates 506 g maybe obtained from transmission features of the RFID tag 512 a or othertype of electronic label.

Using the location coordinates 506 g, in some implementations, anaudible locator module plays audible tones to the individual 502 thatindicate relative distance and/or direction of each object 508, 510 fromthe individual 502. The intensity and directionality (e.g., left/rightbalance or other speaker distribution) of the audible tones, forexample, can be stored as presentation feedback data 506 e of thewearable data collection device 504. Each object 508, 510, further, maybe associated with a particular sound. For example, the ball object 508may be indicated by a bouncing noise, while the clock object 510 may beindicated by a tick-tock noise. Using the audible locator algorithm 548,a blind individual 502 could discover the nature of her environment byreceiving audible feedback representing the depth and breadth of a roomand the location of objects within it by scanning the scene andreceiving audible tone-based feedback from the wearable data collectiondevice 504. Alternatively or additionally, the presentation feedbackdata 506 e regarding locations of the objects 508, 510 can includetactile or haptic feedback. For example, the machine vision languagetutor algorithm 532 a may translate distance and relative position of anobject into vibrational intensity, patterns, and application point(should multiple tactile feedback application points be available uponthe body of the individual 502).

In some implementations, an object tracking software module of themachine vision language tutor algorithm 532 a tracks thethree-dimensional object location during a period of time. For example,tracking of the position of each object within a vicinity may aid ininventory management. During chemistry experiments in a chemistrylaboratory, for example, the object tracking software module maydetermine which laboratory technicians interacted with each of thevarious chemical compounds, pieces of equipment, and other objects withstandardized index elements within the vicinity of the laboratory. Basedupon timestamps associated with object location data 506 f, in oneillustration, the object tracking software module may identify, in someexamples, when particular laboratory technicians interacted with aparticular object, how long a particular object was placed within afreezer, and/or where objects were placed relative to each other in arefrigerated storage area (e.g., on a shelf above or below anotherobject). In other implementations, the object tracking software modulefunctions as a standalone algorithm, not including the language learningand/or graphic enhancement features of the machine vision language tutoralgorithm 532 a.

In some implementations, by analyzing object location data 506 fcross-referenced with one or more of motion tracking data 116 h, videorecording data 116 b and audio recording data 116 a, the machine visionlanguage tutor 532 a (or software tracking module) may identify how theindividual 502 has interacted with a particular labeled object 508, 510.For example, the machine vision language tutor 532 a may identify thatthe individual 502 threw the ball 508 to the right of the clock 510.Furthermore, analysis of the audio recording data 116 a may deriveinformation regarding the level of familiarity of knowledge theindividual 502 has with a particular object, for example throughrecognition of the individual 502 speaking the name of the object.

In some implementations, the level of familiarity, level of comfort,and/or level of discomfort the individual 502 has with a particularobject may be derived through physiological data, such as heart andbreath data 116 e, EMG data 116 i, or EEG data 116 f, described inrelation to FIG. 1A, as well as voice pitch changes (e.g. derived fromaudio recording data 116 a). Furthermore, in some implementations, thewearable data collection device 504 or peripherals in communicationtherewith may collect data regarding skin conductance dynamics, skintemperature dynamics, core temperature dynamics, and other physiologicaldata for use in familiarity analysis.

In some implementations, an object learning software module of themachine vision language tutor 532 a acquires information regardingobjects with standardized index elements, improving in objectidentification such that a labeled object may eventually be identifiedeven when the standardized index element is not visible within the videorecording data 116 b. In some implementations, a portion of the data 116and/or 506 acquired by the wearable data collection device 504 isprovided to a remote analysis system 514. The remote analysis system 514may collect session data 516 provided by the wearable data collectiondevice 504 for analysis by a processing system 518. The remote analysissystem 514, for example, may perform parts of the machine visionlanguage tutor 532 a functionality described above, such as the objectidentification software module, the object tracking software module orthe audible location identifier module.

As illustrated, the processing system 518 includes a learning dataanalysis module 520 for learning to identify objects. The learning dataanalysis module 520, for example, may collect and archive data from anumber of wearable data collection devices in a data archive 522. Thedata archive 522, for example, may include a database or training fileproviding a machine-learning classifier or cascade of classifiers.Further, the data archive 522 may include a database of objectinformation acquired by multiple wearable data collection devices. Thelearning and data analysis module 520, for example, may categorize theobject information. The term “Ball” such as the ball object 508, forexample, can represent a category including yoga balls, beach balls,tennis balls, footballs, soccer balls, etc.

In some implementations, the learning and data analysis module 520recognizes object identifications and categories of objectidentifications based in part upon demographic data collected from eachwearable data collection device. The demographic data, for example, canidentify geographic information and spoken language. Through use ofdemographic data, for example, the learning and data analysis module 520may learn to differentiate between images of European pears and imagesof Asian pears while recognizing each as being a “pear”. Further, thelearning and data analysis module 520 may identify a yellow curvedobject as a banana in the Boston but a plantain in Borneo.

In some implementations, the pool of learned data derived by thelearning and data analysis module 520 is used to refine standardizedindex element extraction methods or object recognition accuracy. Forexample, the learning and data analysis module 520 may collect multipleviews and rotations of a given object to enhance recognition of theobject. Additionally, the learning and data analysis module 520 maycollect many versions of a particular category, such as a ball, mug, ortelephone, and extract features of items and relationships between thefeatures within the category to derive information about the categoryitself (e.g., invariant and variant features and feature-featurerelationships). The learning achieved by the learning and data analysismodule 520, for example, may feed back to the machine vision languagetutor 532 a, allowing the machine vision language tutor 532 a torecognize items and categories of items without requiring machine coderecognition. A portion of this learning may reside in the learningmodule of the machine vision language tutor 532 a rather than with thelearning and data analysis module 520. Refinements to software modules,such as an object identification module, object data presentationmodule, and object location tracking module of the machine visionlanguage tutor 532 a, in some embodiments, are provided as softwareupdates to the wearable data collection device 504 from the remoteanalysis system 514.

The individual 504, in some implementations, provides feedback regardinglabels applied to objects that do not have standardized index elements(or the standardized index element is not visible from the particularview presented within the video recording data 116 b). For example, themachine vision language tutor 532 a may prompt the individual 504 torespond whether a suggested label for an identified object has beencorrectly applied. The wearable data collection device 504 may forwardthe feedback to the learning and data analysis module 520 to aid inrefinement of the automated recognition feature. For example, thelearning and data analysis module 520 may track frequency of incorrectobject identification and evolve better recognition patterns.

The learning and data analysis module 520, in some implementations,includes a meta-analysis feature for deriving rich information basedupon the data collected from a number of wearable data collectiondevices. In some examples, the learning and data analysis module 520 mayanalyze the collected data to determine a set of objects most commonlypresented to individuals using the machine vision language tutor 532 a.At a further level of refinement, the learning and data analysis module520 may identify commonly presented objects by age or age range of theindividual (e.g., toddlers, grade school children, etc.), geographiclocation of the individual, or other classifications of the individualbased upon demographic and/or medical diagnosis information (e.g., asstored within a user profile associated with each individual). Inanother example, the learning and data analysis module 520 may track andanalyze the performance of individuals (e.g., including the individual504) in learning words, phrases, or other information presented by themachine vision language tutor 532 a. The performance analysis may bebroken down into sub-categories, such as performance by operating modeof the machine vision language tutor 532 a (e.g., single word vs. shortphrases, etc.), age range, geographic location, or other classificationsof individuals based upon demographic and/or medical diagnosisinformation.

In some implementations, the single word reading algorithm 532 b of FIG.5B recognizes text being reviewed by the individual 502 wearing thewearable data collection device 504 and highlights particular portionsof the text for the individual 502. The single word reading algorithm532 b, for example, may use one or more optical character recognitionmodules to identify that text has been captured within the videorecording data 116 b. Upon recognition of the text, the single wordreading algorithm 532 b may magnify, brighten, sharpen, or otherwisedraw forth a portion of the text available to the individual 502 withina display region (e.g., heads up display) of the wearable datacollection device 504. Further, the single word reading algorithm 532 bmay adjust a font style or weight, text color, or other aspects of thepresented font to enhance readability and/or draw further attention to aparticular portion of the text. In adjusting the presentation of theportion of the text identified within the video recording data 116 b, insome examples, the single word reading algorithm 532 b may enhancereadability based upon preferences or capacities of the individual 502.For example, the single word reading algorithm 532 b may enhance thetext in a manner which allows the individual 502, having impairedvision, to better read the text. The modifications applied by the singleword reading algorithm 532 b to the rendering of the text, for example,may include adjustment of the presented text to factor in astigmatism ofthe individual 502, partial blindness, color blindness, or othercondition which may frustrate interpretation of the text.

The single word reading algorithm 532 b, in some implementations,selects a portion of the text from a greater body of text (e.g., threelines, five words, etc.) to highlight. The single word reading algorithm532 b may additionally de-emphasize the remaining text within thedisplay of the wearable data collection device 504, for example bydimming, blurring, or otherwise obscuring or partially obscuring theremaining text. In this manner, the attention of the individual 502 isdirected to a portion of the text that has been highlighted or enhancedby the single word reading algorithm 532 b.

The single word reading algorithm 532 b, in some implementations,provides a moving enhancement of the text. For example, to aid in thereading of lengthier text, such as a newspaper article or page of abook, the single word reading algorithm 532 b may provide the individual502 with the opportunity to “read along” by adjusting the portion of theenhancement through an input mechanism of the wearable data collectiondevice 504. The individual 502, in some examples, may provide an audiblecue (e.g., saying “next”), a visual cue (e.g., “dragging” finger alongtext within video recording data 116 b captured by the wearable datacollection device 504), and/or a physical cue (e.g., touching a portionof the wearable data collection device 504 or a peripheral incommunication with the wearable data collection device 504) to signalthe single word reading algorithm 532 b to advance the highlighting to anext portion of the text.

In some implementations, the learning and data analysis modules 520 maylearn a reading speed and/or preferred adjustment style of theindividual 502, allowing the single word reading algorithm 532 b toautomatically adjust and present the text accordingly until signaledotherwise by the individual 502 (e.g., via an input cue as describedabove). For example, the learning and data analysis modules 520 mayidentify that the individual 5022 progresses more quickly through textwhen presented with a serif font than a sans serif font.

In some implementations, the single word reading algorithm 532 b mayparse the text to recognize words and/or phrases, for example matchingthe terms with associated information. In one illustration, through adatabase look-up (e.g., resident to the wearable data collection device504, executed upon a separate computing device in communication with thewearable data collection device 504, and/or implemented within theremote analysis system 514 of FIG. 5A), the single word readingalgorithm 532 b may identify definitions, pronunciations, graphic orvideo illustrations, audio snippets, and other rich informationassociated with an identified word of phrase. The single word readingalgorithm 532 b may then present enhanced information to the individual502 regarding the presented text, automatically or upon selection. In aparticular illustration, the single word reading algorithm 532 bprovides the individual 502 with the opportunity to select a word orphrase within the text for additional information, such aspronunciation, definition, and/or graphic illustration (e.g., what doesa crested gecko look like, what is the pronunciation of “inchoate”, orwhat does “lethargy” mean).

The single word reading algorithm 532 b, in some implementations, may becombined with other algorithms executing on the wearable data collectiondevice 504, such as, in some examples, a bouncing ball reading algorithm532 c or a graphic enhanced vocabulary algorithm 532 d. Similar to thesingle word reading algorithm 532 b, in some implementations, thebouncing ball reading algorithm 532 c presents, to the individual 502,enhanced text as identified within the video recording data 116 b. Theenhanced text, for example, may be superimposed with an attention windowor otherwise selectively highlighted by the bouncing ball readingalgorithm 532 c to identify text for the individual 502 to read. Forexample, a child may interact with the bouncing ball reading algorithm532 c while reading a favorite book. The bouncing ball reading algorithm532 c may present a portion of the text of the book in a highlighted orenhanced fashion, then analyze audio recording data 116 a to identifyaudible terms corresponding to the text on the page. As the child reads,the bouncing ball reading algorithm 532 c may advance the enhancedportion of the text along the page of the book as presented in videodata upon a display region of the wearable data collection device 504.

The bouncing ball reading algorithm 532 c, in some implementations,rewards the individual 502 for correct reading of the text. In someexamples, the bouncing ball reading algorithm 532 c may allocate pointstowards a gaming enhanced interaction (e.g., using a gaming module),illustrate an icon or word of congratulations (e.g., a green checkmarkfor correct reading), or supply audible or tactile feedback identifyingto the individual 502 that the individual 502 read the textsuccessfully.

In some implementations, if the individual 502 struggles withpronunciation of the text or misses or misinterprets words within thetext, the bouncing ball reading algorithm 532 c supplies corrections.For example, the bouncing ball reading algorithm 532 c may correctpronunciation, return to a particular word or phrase to encourage theindividual 502 to try again, or supply a visual, audible, or tactileform of feedback to alert the individual 502 that there were problemswith the reading performance.

The bouncing ball reading algorithm 532 c, in some implementations,includes a reading style learning module (e.g., as part of the learningand data analysis modules) configured to learn, in some examples, theaccent, speech patterns, and other verbal mannerisms of the individual502. For example, the reading style learning module may improve thereading recognition of the bouncing ball reading algorithm 532 c inrelation to the individual 502, such that the bouncing ball readingalgorithm 532 c may recover for a lisp, stutter, or other impedimentwhich may cause greater difficulties in interpreting the vocalization ofthe individual 502 during reading. Further, the bouncing ball readingalgorithm 532 c may be combined with a speech dysfluency coach algorithm540 a (described in greater detail below) to aid in correction of speechdysfluencies identified while interacting with the bouncing ball readingalgorithm 532 c.

Upon conclusion of a portion of reading (e.g., a page, chapter, book,article, etc.), in some implementations, the bouncing ball readingalgorithm 532 c tests comprehension or recall of the individual 502. Forexample, the bouncing ball reading algorithm 532 c may include aquizzing module which correlates information within the text (e.g.,phrases, characters, actions, etc.) with questions for the individual502 to gauge the performance of the individual 502 in reading. In someexamples, the bouncing ball reading algorithm 532 c may verifyunderstanding of a term (e.g., select an appropriate definition),confirm proper identification of a series of actions within a text(e.g., the baker mixed the bread prior to putting the pan in the oven),or identify a particular character (e.g., is Emily a girl, a boy, orcat). The quizzing module of the bouncing ball reading algorithm 532 cmay interoperate with the gaming module, awarding points for correctanswers. The quizzing module, in another example, may feed informationto the learning and data analysis modules 520 to gauge and track thereading level of the individual 502, along with strengths and weaknessesof the reading abilities of the individual 502.

In some implementations, a graphic enhanced vocabulary algorithm 532 dillustrates an image or a visual-sentence action to accompany andtransliterate what is being read. For example, while using the singleword reading algorithm 532 b or the bouncing ball reading algorithm 532c, the reading activity may include visual information appended to thedisplay (e.g., proximate to the text being read) by the graphic enhancedvocabulary algorithm 532 d. In another example, the graphic enhancedvocabulary algorithm 532 d may function in tandem with the machinevision language tutor 532 a to provide image data and/or avisual-sentence action corresponding to an identified object in thevicinity of the individual.

In some implementations, a consonant-slowing speech filter algorithm 534a provides an individual with the opportunity to slow verbal dialoguefor better comprehension. Individuals with autism spectrum disorderoften struggle to hear consonants well. Because of the difficulty withconsonant recognition, boundaries between words may be blurred. Theconsonant-slowing speech filter algorithm 534 a may filter audio datacaptured by the wearable data collection device prior to presentation tothe individual 502 (e.g., via an audio output feature such asheadphones, ear buds, or bone conduction speaker). In the event that theaudio output method is not audio-suppressing (e.g., noise-suppressingheadphones), the output of the consonant-slowing speech filter algorithm534 a may be presented such that it overlays speech the individual isnaturally hearing.

In some implementations, the consonant-slowing speech filter algorithm534 a functions with other modules and algorithms presenting audio datato the individual 502 such that, prior to output, any speech relatedaudio data is filtered to slow consonants for better comprehension bythe individual 502. For example, during review of video traininginformation or presentation of verbal information regarding an objectidentified through the machine vision language tutor algorithm 532 a,the consonant-slowing speech filter algorithm 534 a may be called toslow the consonants of the speech portion of the audio output prior topresentation to the individual 502.

A boundary-enhancing speech filter 534 b, in some implementations,alters audio data containing verbal components to accentuate words andsegment boundaries. In this manner, the boundary-enhancing speech filter534 b may act as an edge-detector or edge-enhancement filter forlinguistic elements. The boundary-enhancing speech filter 534 b mayfilter audio data captured by the wearable data collection device 504prior to presentation to the individual 502 (e.g., via an audio outputfeature such as headphones, ear buds, or bone conduction speaker). Inthe event that the audio output method is not audio-suppressing (e.g.,as in noise-suppressing headphones), the output of theboundary-enhancing speech filter 534 b may be presented overlayingspeech the individual is naturally hearing.

In some implementations, the boundary-enhancing speech filter 534 bfunctions with other modules and algorithms presenting audio data to theindividual 502 such that, prior to output, any speech related audio datais filtered to slow consonants for better comprehension by theindividual 502. For example, during review of video training informationor presentation of verbal information regarding an object identifiedthrough the machine vision language tutor algorithm 532 a, theconsonant-slowing speech filter algorithm 534 a may be called to slowthe consonants of the speech portion of the audio output prior topresentation to the individual 502. Further, the boundary-enhancingspeech filter 534 b may coordinate with the consonant-slowing speechfilter 534 a to both slow consonants and enhance boundaries of speechprior to presentation to the individual 502.

A speech dysfluency coach algorithm 540 a, in some implementations,reviews audio data collected by a wearable data collection device 504 inreal time to identify speech “tics”, filler utterances (e.g., umm, err,etc.), stuttering, and/or other speech dysfluencies. Responsive toidentifying a speech dysfluency, the speech dysfluency coach algorithm540 a may cue the individual 502 using the wearable data collectiondevice 504, for example using a visual, audible, or haptic cue. Uponproviding the cue, the speech dysfluency coach algorithm 540 a mayassess effectiveness of the cue. For example, the speech dysfluencycoach algorithm 540 a may assess whether the cue threw the individual502 off-course (e.g., stammering, excessive pause, starting over with asentence/topic, etc.). Based upon the assessment of effectiveness, thespeech dysfluency coach algorithm 540 a may alter the style of the cuewhen next presenting feedback to the individual 502.

In some implementations, the speech dysfluency coach algorithm 540 atracks progress over time. As a training and management exercise, thespeech dysfluency coach algorithm 540 a may deduct points foridentification of speech dysfluencies, while awarding points forthreshold timeframes of speech patterns without evidence of speechdysfluency. Progress tracking may include, for example, providing areport to a caregiver, medical practitioner, or educator for assessmentincluding information regarding point accrual, types of speechdysfluencies identified, and/or a comparison of frequency of speechdysfluencies over time.

Similar to the speech dysfluency coach algorithm 540 a, in someimplementations, a profanity and colloquialism coach algorithm 540 creviews audio data collected by the wearable data collection device 504in real time to identify usage of profanity and other base or offensivespeech. Additionally, the profanity and colloquialism coach algorithm540 c may monitor gestures of the individual 502 to identify profanegestures made by the individual 502. Based upon identification ofprofane verbal or physical expressions, the profanity and colloquialismcoach algorithm 540 c may cue the individual 502, deduct points, and/ortrack frequency and type of uses and generate progress reports. Unlikethe speech dysfluency coach algorithm 540 a, the profanity andcolloquialism coach algorithm 540 c may modify response based uponcontext (e.g., identification of other members of a conversation,location, tone of the conversation, etc.). For example, the profanityand colloquialism coach algorithm 540 c may provide strict correction inthe school environment when communicating with a teacher, but relaxedcorrection in the home environment when communicating with a friend.

On a broader range, a social acceptability coach algorithm 540 b, insome implementations, reviews audio data collected by the wearable datacollection device 504 in real time to identify topics of conversationthat may not be socially acceptable in the individual's presentenvironment. The social acceptability coach algorithm 540 b, forexample, may identify key words and phrases, as well as densities of keywords in extended speech, to determine topics of conversation that maybe better avoided. The questionable topics of conversation may becross-referenced with a present environment. For example, a topic ofconversation appropriate at the playground may not be as sociallyappropriate at a funeral. Additionally, the social acceptability coachalgorithm 540 b may consider a cultural environment of the individual502 in determining whether a topic of conversation is appropriate. Thecultural environment, in some examples, may include informationregarding ethnicity, race, gender, age group, context (e.g., school,home, family member's residence, etc.), or religion. Similar to thespeech dysfluency coach algorithm 540 a and the colloquialism coachalgorithm 540 c, the social acceptability coach algorithm 540 b mayissue a warning to the individual 502 to cue the individual 402 to ceaseengaging in the present topic of conversation. Further, the socialacceptability coach algorithm 540 b may alert a caregiver or beginrecording depending upon the level of inappropriateness of a topic ofconversation.

A teleprompter algorithm 544, in some implementations, calls upon anumber of the features of other algorithms 532, 538, and 540 to supportthe individual 502 in giving speeches or otherwise engaging in socialinteractions with others. For example, the teleprompter algorithm 544may present a script to the individual 502 in a heads-up display of thewearable data collection device 504. The teleprompter algorithm 544, forexample, may present a portion of the script at a time in a similarmanner as the bouncing ball reading algorithm 532 c. The script, in someexamples, may be a transcript of an actual speech or sociallyappropriate conversations snippets.

In some implementations, a full conversation snippets algorithm 536 a,working in tandem with the teleprompter algorithm 544, accessesarchetype conversation snippets appropriate to a given circumstance. Theconversation snippets, for example, may be stored in a database withinthe wearable data collection device 504 or on another computing devicein communication with the wearable data collection device 504. Inanother example, conversation snippets may be fed to the individual 502through a live coach (e.g., human) feeding conversation snippets to theindividual 502 over a network through the full conversation snippetsalgorithm 536 a. The coach, in some examples, may be a personalconversational assistant, a caregiver, or a colleague. For example, ifthe individual 502 is meeting with a potential business partner, othercolleagues of the individual 502 may attend the discussion through alive video feed established with the wearable data collection device504, similar in manner to the evaluation features described in relationto FIG. 1A. The colleagues may supply information, such as budgetnumbers, time estimates, and other information, to the individual 502through the full conversation snippets algorithm 536 a.

In automatically selecting an appropriate conversation snippet, in someimplementations, the full conversation snippets algorithm 536 a usesfeatures of the social acceptability coach 540 b and/or the personaldistance coach 542 a to identify situational circumstances (e.g., typeof event, location, ages of other members of the conversation, as wellas cultural, racial, religious, or other factors) as well as presentattitudes of the other members of the conversation (e.g., emotional andbody language cues demonstrating a current emotional state of eachmember of the conversation).

Additionally, in some implementations, a sentences and exchangesalgorithm 536 b coordinates with the teleprompter algorithm 544 to parseelements of the conversation, identifying emotional cues within thespeech of the individual 502. While the individual 502 is speaking, forexample, the sentences and exchanges algorithm 536 b may parse audiodata collected by the wearable data collection device for speechelements such as, in some examples, the tone of voice and the ongoinglilt and rhythm (prosody) of the individual's voice, using this analysisto derive verbal emotional cues provided by the individual 502 to theother members of the conversation. In the example of prosody, thesentences and exchanges algorithm 536 b may analyze individual wordchoices, words and phrases used as colored by the greater conversations,and/or characteristics applied to words or phrases (e.g., boldness,formality, familiarity, etc.). Further, based upon analysis of theongoing conversation, the sentences and exchanges algorithm 536 b maypresent one or more cues to the individual 502 through the wearable datacollection device 504. For example, the sentences and exchangesalgorithm 536 b may present an audible cue and/or visual cue to identifya point at which the individual 502 should pause or should emphasis aword while presenting a conversation snippet or speech fed to theindividual 502 by the teleprompter algorithm 540.

In some implementations, the teleprompter algorithm 544 coordinates withthe timing of cultural and conversational gestures algorithm 538 aand/or the performance of cultural and conversational gestures algorithm538 b to prompt the individual 502 to insert appropriate gestures (e.g.,nodding, smiling, etc.) at the appropriate time. Further, the timing ofcultural and conversational gestures algorithm 538 a may prompt theindividual 502 to reduce gesturing, for example upon identifying that alevel of movement of the individual 502 is likely to have a distractingeffect on the other members of the conversation or audience. In someimplementations, the timing of cultural and conversational gesturesalgorithm 538 a may monitor a gaze position of the individual 502,prompting the individual 502 to recycle his gaze through the audienceduring presentation of a speech or to look towards the member of theconversation who is presently speaking.

In some implementations, the teleprompter algorithm 544 coaches theindividual 502 on conversational pace during performance of a speech orwhile in conversation with others. For example, the teleprompteralgorithm 544 may prompt the individual 502, visually and/or audibly, toslow down.

The teleprompter algorithm 544, in some implementations, coaches theindividual 502 on loudness of speech. For example, the teleprompteralgorithm 544 may analyze data captured by a microphone feature of thewearable data collection device 504 to measure the sound level of theindividual's voice. Further, the teleprompter algorithm 544 may adjustits analysis to take into consideration background noise and/or nearnessof other members of the conversation (for example by estimatingdistances using features of the personal distance coach algorithm 542a). Responsive to analysis, the teleprompter algorithm 544 may promptthe individual 502 through the wearable data collection device 504,visually and/or audibly, to adjust speaking volume. In a particularexample, the teleprompter algorithm 544 may present, upon a heads updisplay of the wearable data collection device 504, an icon of a cartooncovering its ears and saying ouch when the individual 502 is speakingtoo loud or a cartoon tilting its ear and cupping its hand when theindividual 502 is speaking too softly.

In some implementations, the individual 502 can invoke the teleprompteralgorithm 544 to practice a speech or impromptu conversational skills.For example, the sentences and exchanges algorithm 536 b may be used toautomatically “respond” to the individual 502 through analysis ofsentences verbalized by the individual 502 within audio data captured bythe wearable data collection device 504 and selection of appropriateresponse conversation snippets based upon the analysis. While theindividual 502 is practicing performance of a speech or practicingconversation skills, the teleprompter algorithm 544 may analyze thevocalizations of the individual 502 to evaluate strengths and weaknessesof a performance. For example, the teleprompter algorithm 544 may invokethe speech dysfluency coach algorithm 540 a to coach the individual 502on avoiding filler utterances during practice. Additionally, whilepracticing a predetermined speech, such as a political speech or linesof a play, the teleprompter algorithm 544 may provide the individual 502with the opportunity to scroll backwards or forwards within the body ofthe speech (e.g., repeat practice of a particular line or section of aspeech prior to continuing to another portion), for example throughfeatures of the bouncing ball reading algorithm 532 c.

FIGS. 6A-6D are flow charts of example methods for augmented realitylearning using a wearable data collection device having capability toobtain one or both of video recording data and electronic label data(e.g., wireless label transmissions such as those described in relationto FIG. 5A regarding standardized index elements). The wearable datacollection device may further have the capability to obtain audiorecording data and/or present audible feedback. Additional capabilitiesof the wearable data collection device may include motion sensors, eyetracking sensors, and head position sensors, such as the hardware andsensors described in relation to FIG. 1A. The motion and/or eye trackingdata, for example, may be used by a method 630 to track the gaze of asubject wearing the wearable data collection device. Methods 600, 610,and/or 630 may be performed by one or more software modules executingupon a wearable data collection device such as the wearable datacollection device 504 described in relation to FIG. 5A. In anotherexample, one or more of the methods 600, 610, and 630 (or portionsthereof) may be executed upon a computing device in communication with awearable data collection device.

Turning to FIG. 6A, in some implementations, the method 600 begins withobtaining video data (602). The video data, for example, may includeimages captured by a head-mounted or otherwise body-mounted camera of avicinity surrounding an individual. The video data may represent thesurroundings of the individual as viewed more-or-less through the eyesof the individual.

In some implementations, the video data is analyzed to identify one ormore standardized index elements (604). The standardized index elementsmay be applied as labels to objects, such as the objects described inrelation to FIG. 5A. In other implementations, the standardized indexelements may include visible markings upon or built into the objects. Infurther implementations, the standardized index elements may includeelectronic signals emitted from one or more objects. The standardizedindex elements, in some examples, may include a two-dimensional barcode,three-dimensional barcode, QR code, radio-frequency identification(RFID) tags, and other machine-readable labels or electronicallytransmitting smart labels.

In some implementations, if a standardized index element is located(606), location coordinates of the standardized index element areprovided for further analysis (608). The location coordinates, forexample, may include two-dimensional coordinates (e.g., within a videoframe reference) or three-dimensional coordinates (e.g., with respect tothe point of capture). Subsequent analysis, for example, may be executedupon a same or different processing system involving a same or differentsoftware module or algorithm. The method 600, for example, may call aseparate software algorithm for analyzing the video data at theidentified location coordinates to extract information from thestandardized index element. In addition to location coordinates, a timestamp of the time of video capture may be provided for further analysis.

In other implementations, instead of or in addition to identifyingstandardized index elements, an object or classification of an objectmay be identified. For example, the video data may be analyzed toidentify features corresponding to various objects. As with thestandardized index elements, the location coordinates of the identifiedobjects may be provided for use by a separate software module,algorithm, and/or computing system. Although described as a linearanalysis, in other implementations, the video data is analyzed inparallel (e.g., using multiple threads) and/or recursively to identifystandardized index elements.

Turning to FIG. 6B, a flow chart illustrates an example method 610 foranalyzing an identified standardized index element to derive objectinformation. In some implementations, the method 610 begins withreceiving the location coordinates of the standardized index element(612). As described in relation to FIG. 6A, the location coordinates maybe supplied from a separate algorithm or module executing upon a same ordifferent processing system. In some implementations, information isextracted from the standardized index element (614). One or morehardware, firmware, or software elements of a wearable data collectiondevice, for example, may be used to scan the video data for thestandardized index element. For example, an RFID scanner feature of awearable data collection device or other machine-vision processes may beused to scan the standardized index element for information. To improverecognition of objects encoded with standardized index elements withinthe vicinity, in some implementations, two or more separate methods maybe used to identifying items. Objects identified using one recognitionmethod may be cross-referenced with the objects identified using thesecond recognition method. In other implementations, audio data and/orwireless transmission data may be reviewed using machine-hearing orother signal processing abilities to identify audible or otherelectronic signals of standardized index elements.

In some implementations, a standardized index element only partiallyidentifiable within the video feed may be read (if readable by one ormore scanning systems) to obtain an object index. Further, if the objectwas previously scanned and recognized, based upon a visible portion ofthe standardized index element, the method 610 may be able to identifythe particular object (e.g., using information in a local database ortraining file entry associated with the object having the standardizedindex element). A shape of the object in combination with a partialstandardized index element, in a particular example, may be used touniquely identify the object.

In some implementations, the information extracted is reviewed for aknown index or other code (616). Each standardized index elementconfigured for use with the method 610, for example, may be embeddedwith a particular identifier (e.g., substring) that is otherwiseunlikely to occur in that particular type of standardized index element,such that the identifier can be used to identify standardized indexelements created for use with the wearable data collection device.Alternatively, the standardized index element may be embedded with asimple indexing term, such as a noun identifying the associated object.

If the standardized index element includes a known index or other code,in some implementations, object information is matched to the registeredcode or indexing term (618). For example, the object code or index maybe applied to a look-up table to derive associated object data regardingthe encoded object. In other examples, the standardized index element isa smart label such as an RFID tag including embedded object data. Inthis circumstance, the embedded object data is extracted from thestandardized index element.

In some implementations, the object information is provided to one ormore active modules configured to utilize the object information (620).The method 610, for example, may call a separate software algorithm forusing the object information to present feedback to an individual.

In some implementations, if the information extracted does not include aknown index or other code (616), the standardized index element isreviewed for identifying information (622). If identifying informationis extractable by the method 610 from the standardized indexing element,in some implementations, the object information is provided to one ormore active modules configured to utilize the object information (620).For example, if a machine-readable code derived from an object can beused to positively identify the object, such as the UPC code upon aproduct, the name of the product may be provided to the one or moreactive modules for use. Further, in some implementations, the object,identified by the machine-readable code, may be added to a database ortraining list of identified objects (e.g., stored within a wearable datacollection device or another computing device in communication with thewearable data collection device).

Turning to FIGS. 6C and 6D, a method 630 uses identified objects topresent information to an individual donning a wearable data collectiondevice. In some implementations, the method 630 begins with receivingobject information matching a standardized index element extracted fromvideo data as well as location coordinates identifying a location of theobject within the video data (632). As described above, the objectinformation and location coordinates may be supplied from a separatealgorithm or module executing upon a same or different processingsystem.

If the object information corresponds to an object which was recentlypresented to the individual (634), in some implementations, the method630 returns to awaiting receipt of additional object information. Inthis manner, if an individual was recently presented with informationregarding the object, the individual is not repeatedly presented withidentical information. A database or log file lookup, for example, mayidentify when (if ever) the object information was last presented. Athreshold time, for example, may be used to determine whether to presentinformation to the individual regarding the identified object.

If the object was not recently presented to the individual (634), insome implementations, a language mode and/or presentation mode isidentified (636). For example, a target language setting (or languagesettings when presenting both a native language and foreign language)may be accessed to determine a language for presentation of any textualand/or verbal feedback presented to the individual. If a languagesetting includes a language not stored within the object data, the termin a stored language (e.g., English) may be provided to a translationmodule (internal to the wearable data collection device or externallyaccessed via a network connection) for translation. Presentationoptions, in some examples, may include a visual text display setting, averbal (audible) presentation display setting, and an associated sound(audible) setting. Other presentation settings can include options oflearning level or information scope, such as a level of vocabulary,whether to use meta-category labels (e.g., object “dog” belongs tocategory “animal”, etc.), and whether to present single terms orsentences.

If one or more visual presentation settings are active (638), in someimplementations, a visual presentation is prepared based upon thepresentation mode and language mode (640). The visual presentation, forexample, may be prepared for overlay upon current video data. Forexample, as described in relation to FIG. 5A, the video recording data116 b may be overlaid with a textual representation of one of thelabeled objects, such as the word “ball” applied upon or over the ballobject 508.

Rather than overlaying with object data, in another example, each theobject may be identified as selectable within presented video data byaugmenting the video data at or proximate to the location coordinates ofthe object. For example, the presentation may colorfully outline theobject, render the object as a cartoon, cause the object to shimmer, orotherwise augment the object to draw the attention of the individual.

In some implementations, if it is determined that the focal point of thevideo data captured after the time of identification of the standardizedindex object has moved (642), the location coordinates are adjusted tocompensate for the movement (644). For example, based upon motion of thehead of the individual donning the wearable data collection device, thecurrent location of the object may be calculated and the placement ofthe graphic overlay of the video data adjusted. Conversely, if theobject was in motion during video capture, motion data associated withthe object may be used to estimate a present position of the objectwithin the video.

In some implementations, the visual presentation is presented at orproximate to the location coordinates within the video data (648). Thepresentation, for example, may be overlaid upon a present video dataframe and caused to display to the user. The user, for example, may seethe visual presentation upon a heads-up display of the wearable datacollection device.

If one or more audio presentation settings are active (650), in someimplementations, audible feedback is prepared for presentation to theindividual (652). The audible feedback, for example, may include a word,sentence, and/or sound associated with the identified object.

In some implementations, the audible feedback is provided to an auditoryoutput system (654). The auditory output system, in some examples, mayinclude a speaker system, bone conduction speaker system, or a tetheredaudio output device (e.g., headphones or ear buds, etc.).

The method 630 continues in FIG. 6D. Turning to FIG. 6D, in someimplementations, the individual is presented with an opportunity toselect an object (656). Selection of an object, in some examples, may beperformed by the individual through an input feature of the wearabledata collection device such as a tap, voice command, gesture, or thoughtpattern.

If an object is selected (656), in some implementations, additionalobject data regarding the selected object is presented (658). Theadditional data, for example, can include a deeper level of information,such as, in some examples, one or more terms associated with the objectused in a grammatically correct sentence, a description associated withthe selected object (e.g., brief encyclopedia-style write-up regardingthe object), or other terms used to describe the object (e.g., a car canfurther be called a vehicle, auto, automobile, etc.). In a particularexample, the additional object data includes a vocalized pronunciationof the name of the object.

Selection of the additional information, in some implementations, maydepend upon an options menu. The menu may include options such assentences, usage guides and tips, long definition, images of alternativeversions of the object or previous exemplars in the world viewed by thewearer.

In some implementations, a response is received from the individual(660). The individual's response, in some examples, can include a vocalresponse (e.g., name of the object or other vocalization that mayrepresent familiarity with the object), a physical response (e.g.,picking up, touching, or otherwise interacting with the object), and/oran emotional response (e.g., an emotional reaction that may be gaugedusing voice reflection analysis of audio recording data and/or analysisof various physiological data collected by the wearable data collectiondevice, as described, for example, in relation to FIG. 1A).

If a response is received from the individual (660), in someimplementations, the response is validated (662). A vocalized responsemay be analyzed to identify familiarity with the object. A physicalresponse, in some examples, may be analyzed to identify a comfort levelthe subject has with the object, dexterity demonstrated regarding use ofthe object, and/or correctness of use of the object (e.g., a ball objectis thrown, not bitten). Further to the example above, the individual mayrepeat the vocalized pronunciation of the name of the object. Theindividual's utterance may be recorded as audio recording data andanalyzed to determine how well the individual pronounced the name of theobject. Validation data, in some implementations, may be recorded to aidin assessment of the individual and/or to track progress of theindividual in interacting with objects within the vicinity (e.g., homeenvironment).

In some implementations, feedback regarding the response is provided tothe individual (664). The feedback, in some examples, may be presentedto encourage a desired reaction to or interaction with the object,discourage an undesired reaction to or interaction with the object,and/or represent relative success in performing a task associated withthe object, such as pronouncing the name of the object. Feedback data,in some examples, can include visual feedback, audible feedback, and/ortactile feedback. In the particular example of representing relativesuccess in performing a task associated with the object, a visualindication of a green check or red “X” presented in a heads up displayof the wearable data collection device may visually represent success orfailure related to the task (e.g., pronouncing the name of the object).Further to the example, in addition to or instead of a visualindication, an audible indication (e.g., fanfare or buzzer) may be usedto provide feedback to the individual. Additional discussion regardingthe use of feedback and selection of styles of feedback is provided inrelation to the method 800 of FIG. 8.

FIGS. 7A through 7C illustrate a flow chart of an example method 700 foridentifying socially relevant events and collecting informationregarding the response of an individual to socially relevant eventsusing a wearable data collection device. The method 700 may be used inthe assessment of an individual's reactions as compared to anticipatedtypical reactions (e.g., from a typical person sharing characteristicswith the subject such as age, sex, developmental stage, etc.). Further,the method 700 may be used in coaching an individual in appropriateresponses to social situations.

The wearable data collection device may be capable of collecting videodata and/or audio data. The wearable data collection device may furtherhave the capability to present audible and/or visual feedback.Additional capabilities of the wearable data collection device mayinclude motion sensors, eye tracking sensors, and head position sensors,such as the hardware and sensors described in relation to FIG. 1A. Themotion and/or eye tracking data, for example, may be used by the method700 to track the gaze of an individual wearing the wearable datacollection device. The method 700 may be performed by a software moduleexecuting upon a wearable data collection device such as the wearabledata collection device 104 described in relation to FIG. 1A or thewearable data collection device 504 described in relation to FIG. 5A. Inanother example, the method 700 may be executed upon a computing devicein communication with a wearable data collection device.

In some implementations, video data and/or audio data are obtained(702). The video data, for example, may include images captured by ahead-mounted or otherwise body-mounted camera of a vicinity surroundingan individual and a second person (e.g., caregiver, family member,evaluator, etc.). The camera may collect video data from the perspectiveof the individual or the second person. Further, a second camera may beused, such that video data represents both the viewpoint of theindividual and the second person. The video data may represent thesurroundings of the individual and/or second person, for example, asviewed more-or-less through the eyes of the individual/second person.The audio data, similarly, captures at least vocalizations between theindividual and the second person, for example via a microphone mountedon the wearable data collection device or separate computing device.

In some implementations, based upon the video data and/or audio data, asocially relevant event is detected (704). The social relevant event caninclude an emotional expression typically evocative of an appropriateresponse by the other party such as, in some examples, smiling,laughing, crying, admonishing in an angry tone, asking a question, usingprofanity, or invoking the name of the other party. In analyzing thevideo and/or audio data for a socially relevant event, emotionalresponses can be characterized by one or more of voice fluctuations,tone, cadence, volume, and prosodic variation of the voice of thespeaker, facial expressions, body language, and hand gestures.Furthermore, emotional responses may be derived, in some embodiments,through collection of physiological data, such as the physiological datatypes described in relation to FIG. 1A (e.g., heart rate, breathingrate, EMG, EEG, etc.). In one example, determining an emotional stateassociated with the socially relevant event includes providing thevarious data described above to a classifier which applies aclassification of emotion and valence.

In some implementations, it is determined whether to adjust formitigating factors (708). The method 700, in some embodiments, reviewscollected data for extenuating circumstances or other characteristicsthat may depress typical emotional response. For example, whileinvocation of the individual's name may typically cause the individualto turn to the attention of the speaker, if the individual is presentlydistracted (e.g., by a television show, loud noises, nearby activity, ordeep concentration in a personal activity) the normal (anticipated)response may be suppressed in the typical individual. Similarly, theindividual may respond differently based upon the emotional state of theindividual prior to the socially relevant event. In some examples,mitigating factors can include whether the individual was excitable,angry, sad, or otherwise emotionally stimulated in a manner that couldaccentuate or depress typical response to the socially relevant event.In some examples, an emotional state identifying module may evaluatevarious physiological data captured by the wearable data collectiondevice and/or peripheral devices in communication with the wearable datacollection device such as, in some examples, heart and breath data 116e, EMG data 116 i, or EEG data 116 f, described in relation to FIG. 1A,as well as voice pitch changes (e.g. derived from audio recording data116 a). Furthermore, in some implementations, the wearable datacollection device or peripherals in communication therewith may collectdata regarding skin conductance dynamics, skin temperature dynamics,core temperature dynamics, and other physiological data for use inemotional state analysis.

If adjusting for mitigation factors (708), in some implementations, astatistically likely normal response, based upon emotional state,external factors, and/or other internal factors (e.g., level ofconcentration on a task), is determined (714). The statistically normalresponse, for example, may be derived from data collected fromeducators, clinicians, and/or physicians regarding behavioral studiesand common emotional response patterns. Otherwise, a normal (desired)response is determined (712), similarly based upon collected dataregarding common emotional response patterns. In other implementations,the method 700 determines both the normal (desired) response and astatistically likely normal response based upon present mitigatingfactors.

In some implementations, based at least in part upon the statisticallylikely normal response and/or the normal response, a desired response isdetermined (716). The desired response, for example, may include aresponse determined to be appropriate to the particular individualand/or reasonable for the particular individual to achieve. The desiredresponse, for example, may be based upon a spectrum of known responsescommon to the particular individual and/or a personality assessment ofthe particular individual.

In some implementations, the actual response of the individual iscompared to the desired response and/or the normal response(s) (718).The comparison may represent a closeness in match between theindividual's actual response and one or both of the desired response andthe normal response. In some examples, the comparison may include apercentage match or numerical (e.g., level) match. The comparison mayrefer, in a particular example, to a numerical value indicating apositive (e.g., overreaction) difference between the normal response andthe actual response or a negative (e.g., suppressed reaction) differencebetween the normal response and the actual response.

In some implementations, data regarding the socially relevant event,actual response and/or comparison data is recorded (720). The wearabledata collection device, for example, may record the data locally (e.g.,in storage built in or directly accessible to the wearable datacollection device) and/or remotely (e.g., accessing a network-basedsystem for collection and later assessment/statistical learning analysisof the data). Furthermore, data regarding emotional state,circumstances, and/or other mitigating factors may be recorded inrelation to the socially relevant event and response thereto.

In some implementations, the method 700 is used for a number ofpurposes. These purposes are described herein as operational modes.Although represented as separate and discrete modes in the illustratedflow chart, alternatively, the method 700 may perform at least a portionof the steps associated with each of a characterization and learningmode 724 and a training and feedback mode 726.

In some implementations, a characterization and learning (724)operational mode is determined (722). In the characterization andlearning (724) operational mode, if no noticeable/noteworthy differenceis discerned between the individual's actual response and at least oneof the desired and normal responses (728), the method 700 returns to thebeginning and continues to obtain video and/or audio data (702). Theconcept of “noticeable difference” may represent a statisticallysignificant comparison value, for example as determined by behavioralexperts, or may be noticeable in some other way or according to someother thresholding than traditional statistical significance.

If, instead, a noticeable difference is identified (728), turning toFIG. 7B, in some implementations, the data record regarding the sociallyrelevant event is flagged as a noticeable detour from a desired ornormal social response (730). In this manner, for example, lateranalysis can incorporate details regarding any failures of theindividual in reacting appropriately to social events.

In some implementations, if physiological data is available (732), thephysiological data is correlated with the social event, actual response,and comparison data. As described above, the physiological data caninclude heart and breath data, EMG data, or EEG data, as well as otherphysiological factors such as, in some examples, metabolic data,neurological signals, chemodynamics signals, and/or central nervousactivity.

In some implementations, if historic data is available (736), one ormore recent atypical behavioral episodes may be correlated with thesocial event data (738). Atypical behavioral episodes, in some examples,can include inappropriate behaviors such as acting-out, extremeemotional fluctuations, and stimming and similar behaviors. Conversely,in some implementations, upon identification of an atypical behavioralepisode, historical records regarding recent social response may bereviewed to identify any common behaviors leading up to atypicalbehavioral episodes. Identification and management of atypicalbehavioral episodes is discussed in greater detail in relation to FIG.11.

In some implementations, the physiological data and/or historic data arereviewed to identify susceptibility of the individual to future atypicalbehavioral episodes (740). As described above, various physiologicaldata captured by the wearable data collection device and/or peripheraldevices in communication with the wearable data collection device suchas, in some examples, heart and breath data 116 e, EMG data 116 i, orEEG data 116 f, described in relation to FIG. 1A, as well as voice pitchchanges (e.g. derived from audio recording data 116 a) may be comparedto common physiological factors leading up to atypical behaviorepisodes. The comparison, for example, can be both objective andsubjective. Objective comparison of physiological data, for example, caninclude comparing the individual's physiological data to that of otherindividuals exhibiting atypical behavioral episodes similar to those ofthe individual and/or other individuals diagnosed similarly to theindividual (e.g., ASD level identification). Subjective comparison ofphysiological data, for example, can include comparing the individual'spresent physiological data to historic physiological data of theindividual that has been flagged as leading to a past atypicalbehavioral episode. The comparison may result in a numeric valueindicative of present relative susceptibility to an atypical behavioralepisode.

Prior to comparison, in some implementations, emotional andphysiological states may be derived from the individual's physiologicaldata. The states, for example, can include one or more of a mentalstate, an arousal level, and an irascibility level. The stateinformation, in turn, may be used to identify a measurement of theindividual's present susceptibility to an atypical behavioral episode.

If the review outcome is indicative a likelihood of an impendingatypical behavioral episode (742), in some implementations, feedbackrelated to anticipation of a potential atypical behavioral episode ispresented (744). In some implementations, a caregiver is alerted to thelikelihood of an impending atypical behavioral episode. For example, thewearable data collection device donned by the caregiver may present anaudible and/or visual warning regarding the likelihood of an impendingatypical behavioral episode and, potentially, an indication of the typeof atypical behavior anticipated (e.g., acting out, stimming, etc.).Furthermore, the caregiver may be prompted with recommendations ofmeasures to take to best prevent, redirect, and/or minimize the atypicalbehavioral episode. In some implementations, the individual is alertedto the likelihood of an impending atypical behavioral episode. Thewearable data collection device donned by the individual, for example,may present an audible and/or visual warning regarding the likelihood ofan impending atypical behavioral episode similar to the warning suppliedto the caregiver. Further, the individual may be prompted withrecommendations of measures to take to minimize or protect against theimpending behavioral episode. The individual, in some implementations,may be presented with feedback designed to divert a pending atypicalbehavioral episode. For example, feedback may be presented via theindividual's wearable data collection device (e.g., visual, audible,tactile, etc.) designed to alter one or more physiological conditionsindicative of a pending atypical behavioral episode. The feedback, in aparticular example, may be designed to calm the emotional state of theindividual or focus the individual's attention to divert from a presentthought pattern. A variety of particular feedback examples follow. Theindividual may be presented with a short episode of a game that hasproven previously to attract the attention of this individual or otherslike this individual. The individual may be encouraged to focus on aparticular sensation and try to eliminate another sensation from mind.The individual may be instructed to chew or crunch on a food or toy thatprovides comfort or inward focus for the individual. In a particularexample, turning to FIG. 7D, a screen shot 760 includes a prompt pane762 encouraging the user to relax alongside an image pane 764 configuredto provide a pleasurable sensory experience for the user.

Beyond feedback, in some implementations, interventions may be providedon behalf of the individual. For example, a caregiver may be notifiedand instructed to provide the individual a timeout moment, a pleasanttoy, a brief instruction, an enjoyable food or other sensory experience.

In some implementations, the intervention includes a pharmacological orelectrical or magnetic form of interaction. For example, theintervention may include triggering of implanted pharmaceuticaldispensers or systems for selective release of medicines (includingpharmacological agents whose absorption can be influenced externallysuch as by radio frequency (RF), light, or other method for impartingenergy). Furthermore, in some implementations, a stimulator device(described in detail below in relation to FIG. 12) may be used toprovide direct intervention via stimulation. For instance, electrical ormagnetic pulses may be administered directly to the individual via astimulator, and the electrical or magnetic pulses may be associated withan instruction or guided behavior that inhibits a potential atypicalbehavioral episode, or it may directly cause said atypical behavioralepisodes to be less likely, for instance by direct neural action orinfluence. The stimulation, for example, may be used to influence braincircuits by triggering a pleasurable or hedonistic response. Othervariations for applying non-invasive effects upon brain functionsinclude, in some examples, transcranial direct-current stimulation(TDCS), transcranial magnetic stimulation (TMS), and radio-frequencyenergy deposition into tissue, energy deposition into tissue such asbrain tissue via radio-frequency oscillations of electromagnetic fields.The magnetic, energy, electrical, and/or pharmaceutical interventionsmay be automated or semi-automated (e.g., supplied upon approval by acaregiver, medical practitioner, or other authorizing individual).Further, the magnetic, energy, electrical, and/or pharmaceuticalinterventions, in some implementations, may be used to provide feedback,such as game feedback, to the individual in other tools describedherein.

At this point, in some implementations, the method 700 may return tostep 702 of FIG. 7A and continue to collect video and/or audio data. Inother implementations, the method 700 may further record presentation offeedback such that later analysis can discern whether a particularfeedback style appears to stem atypical behavioral episodes in theindividual or not.

Turning to FIG. 7C, if the method 700 is performing in the training andfeedback mode (726), in some implementations, if a noticeable/noteworthydifference is discerned between the individual's actual response and atleast one of the desired and normal responses (744) (e.g., as describedin relation to step 728 of FIG. 7A), the individual is directed toperform the desired response (746). In some examples, visual, haptic,and/or audible coaching mechanisms may be used to trigger a desiredresponse from the individual. In a particular example, a funny sound maybe played to invoke a smile or giggle from the individual in response toa socially relevant event that normally invokes pleasure. The video feedof a heads-up display, in another example, may be augmented to highlighta face for the individual to look at or otherwise direct the gaze of theindividual towards a speaker, such as by using a graphic arrowindicating to the individual to turn her head in a particular direction.Further to the example, a video icon of an arrow may “grow” and “shrink”based upon whether the individual is turning away or towards thedirection of the arrow. Additionally, audio or video feedback may spellout to the individual the particular desired behavior to invoke, such asan audible cue directing the individual to “smile now” or a visual cueincluding the text “shake hands”. This functionality, in one example,may be supplied in part using features of the performance of culturaland conversational gestures algorithm 538 b, described in relation toFIG. 5B.

In some implementations, effectiveness of the presented guidance isdetermined (748). For example, based upon recorded video and/or audiodata, the socially relevant event identifier can identify a sociallyrelevant response invoked by the individual and compare the response tothe prompted response. This step and the following steps 748 and 750, inone example, may be performed at least in part by features of the socialacceptability coach algorithm 540 b, described in relation to FIG. 5B.

In some implementations, if the guidance is determined as having beeneffective (748) positive feedback is presented to the individual (750).The feedback, in some examples, can include visual feedback, audiblefeedback, and/or tactile feedback. In the particular example, a visualindication of a green check is presented in a heads up display torepresent success of the subject in following through on the presentedresponse guidance. Furthermore, in some implementations, the feedbackmay include triggering magnetic, energy, electrical, and/orpharmaceutical doses for enhancing pleasure signals of the individual.

Conversely, if the guidance is determined as having been ineffective(748), in some implementations, negative feedback is presented to theindividual (752). In the particular example, a visual indication of ared “X” is presented in a heads up display of the wearable datacollection device to represent failure of the individual in followingthrough on the presented response guidance. Additional discussionregarding the use of feedback and selection of styles of feedback isprovided in relation to the method 800 of FIG. 8.

Turning to FIG. 8, a flow chart illustrates an example method 800 forconditioning social eye contact response through augmented reality usinga wearable data collection device. The method 800, for example, mayincorporate a type of game or virtual reality activity aimed atconditioning a user assessed for ASD to engage in social eye contact.

In some implementations, the method 800 begins with obtaining video data(802). The video data, for example, includes images captured by ahead-mounted or otherwise body-mounted camera of a vicinity surroundingthe user. The video data may represent the surroundings of the user asviewed more-or-less through the eyes of the user.

In some implementations, one or more faces of individuals are identifiedwithin the video data (804). The faces, for example, can include familymembers, social peers, colleagues, or other people in the surroundings.Additionally, in some embodiments, the faces can include animals orinanimate objects, such as a family pet, a therapy dog, or a toy doll.

In some implementations, at least a first face of the one or more facesin captured video data is augmented to draw attention to the face withinthe video output to the user (806). In some examples, the face may beoutlined in colors, overlaid with a shimmer, or caricatured in ananimated fashion to draw the attention of the user. In other examples,silly hair may be applied to an individual identified within the videodata or a distortion field applied to the face region. Alternatively oradditionally, in some examples, background video surrounding the facemay be dimmed, reduced in complexity, or blurred to reduce focus on anyaspects in the video besides the face. In a particular example, afavorite cartoon character may be superimposed upon the face region ofan individual (e.g., in an opaque or semi-transparent manner) within thevideo data to draw the attention of the user to the face of theindividual.

Alternatively, in other implementations, faces may be removed from thevideo output to the user. For example, the face regions of eachindividual may be edited out of the video feed or supplanted with anoverlay (e.g., solid color, animated grayscale noise pattern, etc.).

In some implementations, data is analyzed to identify social eye contactbetween the user and the first face (808). For example, as described inrelation to FIG. 1A, an eye tracking module may analyze eye trackingdata 116 g obtained from a face-directed video capture element of thewearable data collection device to determine when the gaze of the userco-registers with the first face of the video data. In another example,video captured by a wearable data collection device worn by the otherperson is analyzed to determine whether the gaze of the user is directedat the face of the person. Further, in some embodiments, both the userand the other person have donned wearable data collection devices, and astraight line wireless signal, such as a Bluetooth signal, infraredsignal, or RF signal, is passed between the user's wearable datacollection device and the other person's wearable data collectiondevice, such that a wireless receiver acknowledges when the two wearabledata collection devices are positioned in a substantially convergenttrajectory.

In some implementations, reaction of the user to the augmentation styleis assessed and recorded (808). If the augmentation style failed tocatch the user's attention towards the first face, for example, thefirst augmentation style may be recorded as being “ineffective.”Conversely, if the user's attention turned towards the first face, thefirst augmentation style may be recorded as being “effective.” In thismanner, the method 800 may include a learning aspect to identify mosteffective methods of gaining and holding the user's attention.

In some implementations, if co-registration indicative of social eyecontact between the user and one of the faces is not identified (810),an augmentation style is adjusted (812). For example, if the firstaugmentation style included a line surrounding the first face, theaugmentation style may be adjusted to instead apply a jiggling movementto the face. In another example, if the first augmentation styleincluded a black and white caricature version of the face, a secondaugmentation style may include a colorful caricature version of theface. Furthermore, augmentation style of the background scenery may beapplied and/or adjusted.

In some implementations, if co-registration is identified (810),positive reinforcement feedback is provided to the user (814). Positivereinforcement feedback can include audio, visual, and/or tactile(haptic) feedback designed to reward the user for directing attention tothe augmented face. Positive reinforcement feedback may include anenjoyable or celebratory sound, such as a fanfare, cheering, or happymusic. Verbal positive feedback, such as the words “success”, “hooray”,“good job”, or “way to go” may be audibly or visually presented to theuser. The positive reinforcement feedback may include a color, image,animation, or other pleasing visual representation presented, forexample, in the heads-up display of the wearable data collection device.In some embodiments, positive reinforcement feedback includes addingpoints, for example in the form of a heads-up display icon representingaccumulated points, in a game-style interface. Levels of positivereinforcement may vary based upon desirability of reaction. For example,for a brief period of social eye contact, the user may be presented bypleasing sounds or other encouragement. After a threshold period oftime, the positive reinforcement feedback may be enhanced to include anindication of success. For example, any social eye contact may berewarded in part, but social eye contact for at least a threshold periodof time (e.g., one second, three seconds, etc.) may be rewarded withpoints or a more elaborate/celebratory feedback mechanism.

In some implementations, the user's reaction to the positivereinforcement feedback is ascertained and the user's preferencesadjusted accordingly (816). For example, upon presentation of positivereinforcement feedback, if the user maintains social eye contact for thethreshold period of time, the particular positive reinforcement feedbackprovided to the user may be flagged as being effective with the user.For example, points associated with the feedback may be incremented orthe feedback may be promoted within a list of feedback options. If,instead, the user terminates social eye contact with the face prior tothe threshold period of time despite the use of positive reinforcement,the particular positive reinforcement feedback presented may be flaggedas being ineffective with the user. For example, points associated withthe feedback may be decremented or the feedback may be demoted within alist of feedback options. In this manner, the method 800 may learn themost effective manners of positive feedback for the particular user.

In some implementations, assessment of the user's reaction to thepositive reinforcement feedback is ascertained in part by analyzingvarious data associated with the user. For example, levels of pleasureor displeasure with the currently presented feedback may be derived fromreviewing a subject-pointing video recording to review relative pupildilation, eye moistness, or eyebrow position. Further, levels ofpleasure or displeasure may be derived from reviewing subjectphysiological data such as heart rate, breathing rate, or neurologicaldata such as EEG/EMG/EKG data.

If, instead of maintaining co-registration for the threshold period oftime, the user terminates social eye contact with the face (818), insome implementations, negative feedback is provided to the user (820).Negative feedback, for example, may be selected to discourage anundesirable behavior of the user, such as glancing briefly at the facerather than maintaining social eye contact. The negative feedback mayinclude one or more of audible, visual, and tactile feedback. Inparticular examples, an irritating vibration may be applied to a pointon the skin of the user or an annoying noise may be played to the user.

In some implementations, the user's reaction to the negative feedback isascertained and the user's preferences adjusted accordingly (822). Asdescribed above in relation to step 816 regarding positive reinforcementfeedback, similar analysis and promotion/demotion of negativereinforcement mechanisms may be made to learn the most effectivenegative feedback mechanisms to use with the user. Success of negativereinforcement mechanisms, for example, may be based in part upon howquickly the user returns his or her gaze to the face.

FIG. 9 is a block diagram of an example collection of softwarealgorithms 910 and 912 for implementing identification of and gaugingreaction to socially relevant events. Based upon particularimplementations, individual software algorithms 910 and 912 may executeupon a wearable data collection device 904 (or 906), a computing devicein direct communication with the wearable data collection device 904 (or906) such as a smart phone, tablet computer, or smart watch, or acomputing system accessible to the wearable data collection device 904(or 906) via a network connection, such as a cloud-based computingsystem. The subsets of the software algorithms 910 and 912, in aparticular example, may be configured for performance of a softwareapplication developed for assessment and/or training of a subject withASD.

The software algorithms 912 may differ in functionality based uponwhether they are executing upon or in coordination with a wearable datacollection device 904 of an individual 902 or upon or in coordinationwith a wearable data collection device 908 of a caregiver 906. Forexample, the eye motion analysis algorithm 912 g designed for executionupon the caregiver wearable data collection device 908 may analyze eyemotion based upon video recording data capturing the face of theindividual 902, while the eye motion analysis algorithm 912 g mayanalyze eye motion based upon a camera mechanism of the individual'swearable data collection device 904 directed at the face of theindividual 902 (e.g., directed at and capturing substantially the eyeregion of the face of the individual 902). In another example, a headmotion analysis algorithm 912 a, designed for execution upon thecaregiver wearable data collection device 908, may analyze movements ofthe head of the individual 902 based upon recorded video data of theindividual 902, while the head motion analysis algorithm 912 a designedfor execution upon the individual's wearable data collection device 904may analyze movements of the individual's head based upon one or moremotion sensors built into the individual's wearable data collectiondevice 904. Further, the software algorithms 910 are unique to providingfeatures for the individual 902.

The software algorithms 910 and 912, in some examples, may be used toperform portions of method 700 described in relation to FIGS. 7A through7C, method 800 described in relation to FIG. 8, and/or method 1000described in relation to FIG. 10A.

Further, the software algorithms 910 may be used to supportfunctionality of one or more software algorithms designed as learningtools or behavioral management aids for the subject 902. For example, insome implementations, a timing of cultural and conversational gesturesalgorithm 538 a (illustrated in FIG. 5B) may use the body languageidentifier 910 a to analyze performance of cultural and conversationalgestures by the individual 902. The cultural and conversational gesturesalgorithm 538 a may provide the individual 902 with coaching andtraining on the timing and appropriateness of gestures such as, in someexamples, handshake styles, bows, nods, smiles, and hand and armgestures during speech. Through the emotion identifier 910 e, forexample, the cultural and conversational gestures algorithm 538 a mayidentify that the caregiver 906 is smiling at the individual 902. Anappropriate response would be to smile back. The subject physio analysisalgorithm 910 g may assess the emotional state of the individual 902and/or determine if the individual 904 is already smiling. The promptresponse algorithm 910 c may be invoked by the cultural andconversational gestures algorithm 538 a to prompt the individual 902 tosmile. Upon recognition of a smile of the individual 904, further to theexample, the present feedback algorithm 910 f may be invoked to providepositive feedback to the individual 902.

In some implementations, the cultural and conversational gesturesalgorithm 538 a of FIG. 5B may coordinate with a performance of culturaland conversational gestures algorithm 538 b of FIG. 5B to train theindividual 902 in proper performance of gestures involving largemotions. The performance training, in some examples, may be used tocoach the individual 902 in proper performance of bowing at proper depthwith proper head angle, dancing postures, distress signals, signlanguage, and other non-verbal communication signals. In a particularexample, turning to FIG. 5C, a screen shot 550 illustrates an exampleuser interface for coaching an individual in performing a bow. An imagepane 552 contains an illustration of an avatar performing a bow movementwith a textual label “perform a bow”, while a coaching pane 554 includesboth a message 556 “bend forward keep the ball in the track” as well asan animated illustration 558. In operation, as the individual performsthe bow, a ball icon portion of the animated illustration 558 will movewithin the image pane 552 according to sensed movements of theindividual's head (e.g., based upon data provided by one or more motionsensing devices incorporated into or in communication with the wearabledata collection device). If the individual maintains the ball iconportion of the animated illustration 558 substantially following a pathportion of the animated illustration 558, the individual's body willappropriately perform the gesture of the bow. In other implementations,additional sensor data captured from sensors upon the individual's bodymay be analyzed to validate positioning and motion corresponding to themotion of the head of the individual such as, in some examples, a motionsensor attached to a wrist-mounted device validating that at least oneof the individual's hands is positioned at his or her side. Althoughillustrated as a two-dimensional coaching animation, in otherimplementations, the visual display may present a three-dimensionalanimated graphic for guiding the individual through proper performanceof the gesture. Further, in other embodiments, the avatar icon may bereplaced by an animated illustration or video demonstration of thegesture.

Returning to FIG. 9, aspects of the cultural and conversational gesturesalgorithm 538 a, in some implementations, are used to coach theindividual 902 in martial arts movements and techniques, yoga postures,role-playing game and re-enactment motions, fighting or defensetechniques, and other controlled physical gestures. In furtherimplementations, aspects of the cultural and conversational gesturesalgorithm 538 a are used to provide quantification and feedback foranomalous body motions such as in dystonia or Parkinson's Disease andHuntington's Disease or motor ticks. Similar to the cultural andconversational gestures algorithm 538 a, the performance of cultural andconversational gestures algorithm 538 b may coordinate with the bodylanguage identifier algorithm 910 a. For example, the cultural andconversational gestures algorithm 538 a may invoke the body languageidentifier algorithm 910 a in support of identifying opportunities forperforming a large motion gesture, and the cultural and conversationalgestures algorithm 538 a, responsive to identifying an opportunity, mayinvoke the performance of cultural and conversational gestures algorithm538 b to coach the individual 902 in performing the gesture.

Although the cultural and conversational gestures algorithm 538 a andthe performance of cultural and conversational gestures algorithm 538 bare described in relation to interactions with another person, in someimplementations, the individual 902 may invoke the algorithms 538 aand/or 538 b for practice mode training in cultural and conversationalgestures.

In some implementations, a personal distance coach algorithm 542 a ofFIG. 5B provides the individual 902 with a tool for coaching appropriatedistance to maintain when interacting with another person, such as thecaregiver 906. The personal distance coach algorithm 542 a, for example,may review video data such as video recording data 116 b described inrelation to FIG. 1A to estimate distance between the individual 902 andanother person. For example, the personal distance coach algorithm 542 amay estimate distance based upon depth cues and parallax cues in thevideo recording data 116 b. In another example, a signal transmittedbetween the individual's wearable data collection device 904 and thecaregiver's wearable data collection device 906 may be used to measure apresent distance. In a further example, distance may be estimated basedupon reflection of signals using a laser or sound-based system of thewearable data collection device 904.

In some implementations, the emotion identifier module 910 e maycontribute to assessment of appropriate distance by gauging a level ofcomfort of the person communicating with the individual 902, such as thecaregiver 906. In one example, the level of comfort of the personcommunicating with the individual 902 may be based upon an estimatedemotional state of the other member of the interaction by invoking theemotion identifier algorithm 910 e. In another example, the level ofcomfort of the person communicating with the individual 902 may be basedupon a posture of the other member of the interaction by invoking thebody language identifier 910 a.

The personal distance coach algorithm 542 a, in some implementations,factors in the distance between the individual 902 and the other memberof the interaction, the estimated emotional state and/or posture cues ofthe other member of the interaction, and, potentially, informationrelated to cultural norms (e.g., geographic, racial, religious, etc.) todetermine appropriateness of the current personal distance. The personaldistance coach algorithm 542 a may invoke the prompt response algorithm910 c to prompt the individual 902 to adjust a present distanceaccordingly.

A turn-taking algorithm 542 b of FIG. 5B, in some implementations,monitors conversation and calculates a relative amount of time that theindividual is contributing to a conversation in relation to the amountof time each other member of the interaction is speaking. Individualsdiagnosed with ASD are frequently quiet and remiss to contribute toconversations, while other individuals diagnosed with ASD will talk onat length without providing opportunity for others to contribute to thediscussion. Through reviewing audio data collected by the individual'swearable data collection device 904, such as audio recording data 116 adescribed in relation to FIG. 1A, the turn-taking algorithm 542 b mayprompt the individual 902 a to speak up or, conversely, to politelypause to allow another member of the conversation to jump in. Further,the turn-taking algorithm 542 b may monitor appropriate turn-takingduring a period of time, tracking progress of the individual 902.

Turning to FIG. 5D, in some implementations, the turn-taking algorithm542 b presents visual feedback, such as the feedback user interfacepresented within a screen shot 560. As illustrated in the screen shot560, a topic pane 562 contains an illustration of a speech bubble iconwith the textual label “Share!”, while a feedback pane 564 includes botha message 566 “Remember to take turns in conversation” as well asstatistical feedback 568 representing a percentage time that theindividual has dominated the conversation (e.g., illustrated as 85% andlabeled “your speaking time”). The screen shot 560, for example, may bepresented within a heads up display of a wearable data collection deviceto prompt a user to take turns in conversations with other members ofthe conversation.

Returning to FIG. 9, in some implementations, the turn-taking algorithm542 b generates a report regarding the individual's progress inconversational turn-taking. The report, for example, may be generated ona periodic basis and supplied to a caregiver, medical practitioner,educator, or other person tasked with assessing the progress of theindividual 902.

FIG. 10A is a flow chart of an example method 1000 for identifying andpresenting information regarding emotional states of individuals near auser. Individuals living with ASD frequently struggle with identifyingand reaction appropriately to emotional states of others. The method1000 can support understanding by an ASD individual of the emotionalstates of those around them and appropriate response thereto throughautomated identification of emotional states of nearby individuals.

In some implementations, the method 1000 begins with obtaining videodata (1002). The video data, for example, may include images captured bya head-mounted or otherwise body-mounted camera of a vicinitysurrounding a user. The video data may represent the surroundings of auser as viewed more-or-less through the eyes of the user. In oneexample, the video data is video recording data 116 a captured by thewearable data collection device 104, as described in relation to FIG.1A.

In some implementations, one or more individuals are identified withinthe video data (1004). The individuals, for example, can include familymembers, social peers, colleagues, or other people in the surroundings.Additionally, in some embodiments, the individuals can include animals,such as a family pet or a therapy dog. For example, as illustrated inFIG. 10B, an individual 1022 is identified within video data, asillustrated in a screen shot 1020.

In some implementations, for each individual identified, body languageis analyzed to identify the emotional state of the individual (1006).For example, an emotional identification and training module may reviewan individual's posture, including head position, arm position, and handgestures or other gestures (e.g., hugging, self-hugging, cheek stroking,head scratching, head holding, high-fiving, fist-bumping, pattinganother on the shoulder) for evidence of body language associated with aparticular emotion. In another example, the emotional identification andtraining module may review an individual's facial expression, includingmouth shape, eyebrow position, pupil dilation, eye moistness, and otherfacial cues regarding emotional state. Turning to FIG. 10B, for example,the emotional identification and training module has identified both aface (designated by a focus frame 1024 a) of the individual 1022 and amouth position 1026 (designated by a focus frame 1024 b) of theindividual 1022, as illustrated in an analysis pane 1026. Returning toFIG. 10A, the emotional identification and training module may alsoreview body dynamics such as, in some examples, trembling, bouncing,shaking, rocking, or other motions associated with emotional state.

If audio data is available (1008), in some implementations, the audiodata is analyzed for emotional cues (1010). For example, the emotionalidentification and training module may extract audio associated withverbalizations of a particular individual identified within the videorecording data. The audio may be reviewed for tone, volume, pitch,patterns in pitch (e.g., sing-song, questioning, etc.), vocal tremors,sobbing, hiccupping, laughing, giggling, snorting, sniffing, and otherverbalizations and/or intonations that may be associated with emotionalstate. In some implementations, the emotional identification andtraining module may further identify one or more emotional words orphrases within the audio data.

In some implementations, the audio-derived emotional cues are applied tothe identified emotional state(s) to refine the emotional state of atleast one individual (1012). For example, if the emotional state of theindividual, based upon video analysis alone, suggested two or morepotential emotional states, the audio-derived emotional cues may be usedto promote or demote the various options to identify a most likelyemotional state candidate. In other implementations, for example if theaudio-derived emotional cues are more reliable because the video isobscured or the individual is not facing the camera, the audio-derivedemotional cues may be used as primary reference or sole reference todetermine the emotional state of at least one individual.

In some implementations, information regarding the emotional state of atleast one individual is presented to a user (1014). For example, afeedback algorithm may augment the video feed of a heads-up display of adata collection device to overlay a description of the emotional stateof the individual, such as the word “irritated” floating above theindividual's head or a simplified cartoon icon representing an emotionalstate such as bored, happy, tired, or angry may supplant theindividual's face in the heads-up display or hover hear the individual'sface within the heads-up display. As illustrated in the screen shot 10B,for example, an icon 1028 representing the emotional state of theindividual 1022, as well as a label 1029 (“happy”), are presented withinthe analysis pane 1026. Alternatively or individually, a term orsentence for the emotional state may be presented audibly to the user,such as “mom is happy.” Further, audio or video feedback may spell outto the user the particular response behavior to invoke, such as anaudible cue directing the subject to “smile now” or a visual cueincluding the text “nod your head and look concerned.” If the individualis an animal, the user may be presented with verbal and/or audiblewarnings, such as “may bite” or “back away”.

In some implementations, rather than presenting an emotional state ofthe individual, the application may take a form of a game, where theuser is presented with a multiple choice selection of three potentialemotional states. In this manner, the user may be quizzed to pay closeattention to learning physical and audible cues identifying emotionalstates. Further, based upon the user's responses, an emotional stateawareness tracking module may learn which emotional states are difficultfor the user to identify or whose emotional states are difficult for theuser to identify. For example, the user may have difficulty recognizingemotional states of bearded men. To aid in recognition, feedback to theuser may include hints for identifying particular emotional states, suchas “raised eyebrows indicate surprise”. Turning to FIG. 10C, forexample, a screen shot 1030 including the individual 1022 includes a setof selectable emoticons 1032, were emoticon 1032 a represents a happyemotional state and emoticon 1032 b represents a surprised emotionalstate. The user may select one of the emoticons 1032 (e.g., through aninput device of a wearable data collection device such as a tap, headmovement, verbal command, or thought pattern). The game may then presentfeedback to the user to correct or congratulate the user, based upon aselected emoticon 1032.

Although described in a particular series of steps, in otherimplementations, the method 1000 may be performed in a different order,or one or more steps of the method 1000 may be removed or added, whileremaining in the spirit and scope of the method 1000. For example,rather than analyzing live video data and presenting information relatedto emotion, in some implementations, the method 1000 may be adjusted topresent a review exercise incorporating images of people that theindividual interacted with recently (e.g., in the past hour, day, week,etc.). In a familiar faces review exercise module, for example, aspectsof the method 1000 may be used to quiz the individual on emotionalstates represented by images or short video segments of one or morefaces identified in video data captured by the wearable data collectiondevice. By using short video segments rather than still images, forexample, the familiar faces review exercise module may allow theindividual to derive emotional cues from body language, vocalizations,and other additional information.

FIG. 11 is a block diagram of an example system 1100 for identifying andanalyzing circumstances surrounding adverse health events and/oratypical behavioral episodes and for learning potential triggersthereof. The system 1100 may analyze factors surrounding the onset ofadverse health events and/or atypical behavioral episodes to anticipatefuture events. The factors may include, in some examples, dietaryfactors, fatigue, light sensitivity, noise sensitivity, olfactorysensitivity, and prescription and/or over-the-counter drug consumptionpatterns. Adverse health events, for example, may include migraineheadaches, epileptic seizures, heart attack, stroke, and/or narcoleptic“sleep attacks”. Particular individuals may be monitored for adverseevents related to known health conditions, such as individuals incongestive heart failure or in presence of aneurysm, individualsrecovering from stroke, or individuals suffering from cardiac disease,diabetes, or hypo/hypertension. Further, individuals may be monitoreddue to psychiatric conditions such as panic disorders. Atypicalbehavioral episodes may include, in some examples, swings inmanic-depressive behavior or bipolar behavior, emotional outburststriggered by post-traumatic stress disorder (PTSD), and acting out orstimming episodes related to ASD.

An individual 1102 wears or otherwise carries a data collection device1104, such as the wearable data collection device 104 or 108 describedin relation to FIGS. 1A and 1B. In further examples, the data collectiondevice 1104 may be incorporated in a general purpose personalelectronics device such as a smart phone, tablet computer, or smartwatch or in a specialized health and fitness computing device such as aFitbit® wireless activity monitor by Fitbit, Inc. of San Francisco,Calif. The data collection device 1104 is configured for collection ofvarious data 116, including, in some illustrated examples, audiorecording data 116 a, video recording data 116 b, EEG data 116 f, EMGdata 116 i, heart and breathing data 116 e, motion tracking data 116 h,and eye tracking data 116 g, as discussed in relation to FIGS. 1A and1B. Furthermore, in some implementations, the data collection device1104 may be configured to collect temperature monitoring data 1106 a,including a skin or body temperature of the individual 1102 and/orambient temperatures of the area surrounding the individual 1102. Insome implementations, the data collection device 1104 may be configuredto collect light monitoring data 1106 b, for example as derived from acamera device or simpler light sensor. Scent monitoring data 1106 c mayidentify various fragrances in the vicinity of the individual 1102.Enhanced physiological data monitoring of the data collection device1104, in some examples, may include blood dynamics and chemistry data1106 d (pulse oximetry, blood flow or volume changes, etc.), skindynamics data 1106 e (galvanic skin response and skin conductanceresponse measurements, etc.), and vestibular dynamics data 1106 f usedto monitor the movements of the individual 1102 to gauge whether theyare standing upright versus falling or wobbling and gyrating, such as ahorizon monitor in combination with a motion monitor.

Data 1108 collected by the wearable or portable data collection device1104 (and, potentially, data collected by peripheral devices incommunication with the data collection device 1104), in someimplementations, are used by a number of algorithms 1110 developed toanalyze the data 1108 and determine feedback 1112 to provide to theindividual 1102 (e.g., via the data collection device 1104 or anothercomputing device). The algorithms 1110 may further generate analysisinformation 1114 to supply, along with at least a portion of the data1108, to learning engines 1118. The analysis information 1114 and data1108, along with learning information 1120 generated by the learningengines 1118, may be archived as archive data 1122 for future use, suchas for pooled statistical learning. The learning engines 1118,furthermore, may provide learned data 1124 and, potentially, othersystem updates for use by the data collection device 1104. The learneddata, for example, may be used by one or more of the algorithms 1110executed upon the data collection device 1104. A portion or all of thealgorithms 1110, for example, may execute upon the data collectiondevice 1104. Conversely, in some implementations, a portion or all ofthe algorithms 1110 are external to the data collection device 1104. Forexample, certain algorithms 1104 may reside upon a computing device incommunication with the data collection device 1104, such as a smartphone, smart watch, tablet computer, or other personal computing devicein the vicinity of the individual 1102 (e.g., belonging to a caregiver,owned by the individual 1102, etc.). Certain algorithms 1110, in anotherexample, may reside upon a computing system accessible to the datacollection device 1104 via a network connection, such as a cloud-basedprocessing system.

The algorithms 1110 represent a sampling of potential algorithmsavailable to the data collection device 1104. The algorithms 1104 mayvary based upon the goal of a particular implementation. For example, afirst set of algorithms may be used to anticipate migraine headaches,while a second set of algorithms are used to anticipate ASD-relatedacting out events. Basic to anticipation of events or atypical behaviorepisodes is an event identifier algorithm 1110 a, configured torecognize occurrence of an adverse event or episode. Data collected bythe data collection device 1104 immediately leading to and during theevent identified by the event identifier algorithm 1110 a, for example,may be presented to the learning engines 1118 for review and analysis.

Based upon data collected regarding the individual 1102 and, optionally,additional individuals having the same disorder and potentially sharingsimilarities of symptoms, the learning engines 1118 may derivecorrespondence between events and one or more corresponding factors.Many of the algorithms 1110 are designed to identify factors which maycontribute to one or more health events. For example, an activityidentification algorithm 1110 d identifies activities the individual1102 is engaged in such as, in some examples, driving, watchingtelevision, eating, sleeping, bicycling, working out at a gym, workingat a computer, reading a book, and tooth brushing. The activityidentification algorithm 1110 d, in some implementations, providesinformation to a fatigue analysis algorithm 1110 e which monitors sleeppatterns and/or other symptoms of fatigue (e.g., skin temperature data1106 a, EEG data 116 f and/or EMG data 116 i, heart and breathing data116 e, etc.).

Certain algorithms 1110, in some implementations, are designed tomonitor consumption factors. For example, a stimulant consumptionidentification algorithm 1110 b may identify consumption of caffeinatedbeverages, such as coffee and soda, while a dietary intakeidentification algorithm 1110 f may identify consumption of varioustypes of foods. The stimulant consumption identification algorithm 1110b and/or the dietary intake identification algorithm 1110 f, in someimplementations, identifies food “objects” through data learned by thelearning and data analysis modules 520 described in relation to FIG. 5Atowards object identification. For example, label scanning capabilitiesas described in relation to object identification in FIG. 5A may be usedto identify packaged food items (e.g., bottles of soda, etc.) andidentify ingredients within packaged food items which may prove to betriggers (e.g., aspartame, monosodium glutamate, etc.). Further, theprescription intake identification algorithm 1110 n may use one or morelabel scanning capabilities, described in relation to FIG. 5A, toidentify prescription or over-the-counter drug consumption.

In monitoring consumption factors, in some implementations, the learningengines 1118 may include a dietary intake analysis module for tracking(or estimating) consumption factors such as, in some examples, calories,vitamins, minerals, food category balance, fats, sugars, salt, and/orfluid volume. Based upon video recording data 116 b, for example, thedietary intake identification algorithm 1110 f may estimate (fromrelative sizes of items within an image) a portion of various foodsconsumed by the individual 1102. For example, the dietary intakeidentification algorithm 1110 f may recognize, through label scanning,dietary intake analysis of a prepackaged food item. Additionally, thedietary intake identifier may recognize the consumption of an apple. Alearning engine 1118 may correlate a medium-sized apple with aparticular intake analysis, as well as logging the apple as belonging tothe fruits food group.

Food intake data collected by the dietary intake identifier 1110 f andanalyzed by one of the learning engines 1118, in some implementations,may be provided to the individual 1102 via feedback 1112, for example,to aid in healthy eating choices and weight loss monitoring. In anotherexample, food intake data may be provided to a caregiver, personalcoach, or health professional for review in relation to treatment of ahealth condition, such as hypertension.

In some implementations, a portion of the algorithms 1110 are designedto monitor triggering factors such as, in some examples: loud,irritating, or potentially frightening noises via a noise intensityanalysis algorithm 1110 i; strobing, intense, or unusually coloredambient light via a light intensity analysis algorithm 1110 i; subtlebut potentially aggravating noises via a background noise analysisalgorithm 1110 k, and strong or potentially evocative scents via a scentanalysis algorithm 1110 g (e.g., fed by scent data 1106 c collected by ascent monitor). In the example of ASD, a potential trigger includesvowel-consonant boundary analysis to identify when nearby speakers maybe mumbling or slurring words. The vowel-consonant boundary analysis,furthermore, can indicate the state of the individual 1102, such ascontributing to fatigue analysis 1110 e or identifying a drugged state(e.g., building into the prescription intake identifier 1110 n).

In some implementations, a portion of the algorithms 1110 are designedto monitor for physiological factors leading to an event. For example, avocalization analysis algorithm 1110 o may identify voice fluctuationpatterns that may later be identified (e.g., by the learning engines1118) to commonly precede adverse health events. EMG data 116 i and/orEEG data 116 f may further be analyzed by the learning engines 1118 toidentify neurological data patterns commonly preceding events.Algorithms 1110 may then be designed to identify the advent of suchneurological data patterns.

In some implementations, rather than collecting EMG data 116 i and/orEEG data 116 f, the data collection device 1104 is designed toindirectly monitor cardiovascular dynamics to reveal underlyingphysiological functions. The core principle is the following: when theheart beats, an impulse-wave of blood courses through the body via thevasculature. As the impulse travels through the body, the body actuallymoves, physically. Certain parts, such as extremities, move in morepronounced ways. The head, for instance, moves in a bobble fashion,perhaps in part because the exquisite joints of the neck allow manydegrees of freedom of motion and because the head is weighty andreceives a large amount of the force of traveling blood and becausemuscles in the neck serve to stabilize the head and may causereverberations with each beat. This may result in particularlypronounced head motions in the case of anomalous heart beats, such as indisease or sudden exertion, if the musculature evolved and learned toaccommodate for healthy and statistically more frequent heart beat andpulse-wave dynamics Specific heart defects or types of cardiac diseasetypically result in anomalous head motions. For instance, aforward-backward head bob can indicate one type of heart problem while aside-to-side head bob can indicate another.

A portion of the algorithms 1110, thus, may be designed to indirectlymeasure physiological dynamics of the body, such as heart rate andcardiovascular dynamics by means of motion sensors, such as one or moreaccelerometers, gyroscopes, magnetometers, gravity sensors, and/orlinear accelerometers. The motion sensors may be positioned at strategicpoints on the body of the individual 1102 such as on the head or atother extremities. Various configurations and deployments of motionsensors may include standalone motion sensors, one or more motionsensors incorporated into a separate device, and one or more sensorsincorporated into the wearable data collection device 1104. The wearabledata collection device 1104, for example, may be head-mounted,incorporating a number of sensors feeding data to a small motionanalysis algorithm 1110 m to derive cardiovascular dynamics information.The small motion analysis algorithm 1110 m, for example, may be designedto measure motions of the body, especially body parts distant from theheart, that are secondary to actual heart (muscular) motions. Forexample, the small motions may relate to flow dynamics of blood, impulsewaves in the vascular system related to heart contractions (healthy oratypical), motions related to muscular contractions in the bodyfunctioning as part of bodily systems to control and counteractpulse-related motions (e.g., such as pulses in the neck region, temples,etc.), and/or other related motions.

In some implementations, a body motion analysis system includes numberof algorithms 1110 as well as one or more learning engines 1118 toextract physiological-motion data and to interpret thephysiological-motion data. For example, the small motion analysisalgorithm 1110 m separates motions related to relevant physiologicalevents (such as heart beats or breaths, among other possiblephysiological target motions) from other motions (such as those fromwalking or gestures). An additional algorithm 1110 or learning engine1118 component of the body motion analysis system, further to theexample, receives physiological event motion data from the extractioncomponent and operates on the information, in order to revealphysiological information such as heart dynamics or breathing dynamics.

In a simple illustrative example, the wearable data collection device1102 includes an inertial measurement unit (IMU) sensor system such asan accelerometer and gyro complex, integrated directly with hardware andsoftware drivers. While worn by the individual 1102, the sensor systemphysically moves with the head with the pulsatile motion of the bloodcoursing through, e.g., the carotid and cerebral arteries (the“ballistocardiogram”). The sensor system, further to the example, may bedirectly attached to a sensor driver complex including a printed circuitboard with components that drive the IMU and acquire data from it, ananalysis unit, and a power source.

In some implementations, to allow the data collection device 1104 tocollect physiological data based upon small motions, the individual 1102first calibrates the data collection device 1104 to identify the pulseor breathing patterns through motion data. For example, if the datacollection device 1104 includes a portable personal electronics devicesuch as a smart phone, the individual 1102 may hold the data collectiondevice 1104 at arm's length while aiming a camera lens at his face todetermine pulse, and calibrate motion-based one. For the wearable datacollection device 1104 with a face-presenting camera device, in anotherexample, a calibration mode may include standing quietly and still whilethe data collection device 1104 calibrates based on motions identifiedvia the face-presenting camera.

In addition to motion sensors, other sensors incorporated into the datacollection device, in some implementations, are used to derive smallmotion data. For example, the small motion analysis algorithm 1110 m mayanalyze video recording data 116 b to interpret deflections of ahead-mounted camera as motions indicative of heartbeat, or sinusoidalarc motions as breathing. In another example, a laser sensor, forexample incorporating interferometry readings, may be used to sensesmall motions. A light sensor collecting light monitoring data 1106 b,for example, may provide interferometry data for the analysis.

In some implementations, additional data sources may be used to infercardiovascular dynamics data. For example, a heat fluctuation analysisalgorithm 1110 l may measure heat fluctuations related to small motionsof the body. These heat fluctuations, for example, may be related tocardiovascular or other dynamics. Heat fluctuations may be measured byany number of available heat measurement devices for surface and radiantheat, including commercially available thermometers, thermistors,digital heat sensors, and other temperature sensors as well as devicesor elements thereof having thermoelectric and pyroelectric materialsand/or generators. When incorporating thermoelectric and pyroelectricmaterials, the wearable data collection device 1104 may further beconfigured to collect heat energy as a supplemental source of power forcharging a battery system of the wearable data collection device 1104and/or one or more peripheral devices. In an example configuration, thewearable data collection device 1104 may include a heat measurementdevice such as a far-infrared camera or sensor mounted proximate to theface of the individual 1102 and separated by a small distance (e.g.,mounted on a short stalk extending from the wearable data collectiondevice 1104), with a line of sight to the facial skin or other bodilyskin. In another example, a small noise analysis algorithm 1110 p may“listen” for breathing or other small sounds associated with heart beator pulse, as well as sounds associated with small body motions thatresult from the pulse and/or breathing. An eye motion analysis algorithm1110 c, in a further example, may analyze blinking, eyelid dynamics,and/or eye movement dynamics.

Using the data collected by the small motion analysis algorithm 1110 m,eye motion analysis algorithm 1110 c, heat fluctuation analysisalgorithm 1110 l, and/or small noise analysis algorithm 1110 p, in someimplementations, one or more learning engines 1118 may infer a varietyof physiological data. The physiological data can include heart dynamicssuch as, in some examples, heart rate, heart rate variability, QRScomplex dynamics, heart beat amplitude, or murmur, and fibrillation.Further, the physiological data can include breathing dynamics such asbreathing depth, breathing rate, and identification of yawning (e.g.,potentially feeding back to the fatigue analysis algorithm 1110 e).Other possible extensions include gut dynamics, body motions associatedwith seizures or autistic tantrums, and cerebral blood flow dynamics(e.g., providing insight into brain dynamics).

Using the data collected by the small motion analysis algorithm 1110 m,eye motion analysis algorithm 1110 c, heat fluctuation analysisalgorithm 1110 l, and/or small noise analysis algorithm 1110 p, in someimplementations, one or more learning engines 1118 may infer informationrelated to various unwellness conditions or health states. Theunwellness conditions can include, in some examples, neurodegenerativeconditions such as Huntington's Disease, Alzheimer's Disease,Parkinson's Disease, prion diseases, other spongiform encephalopathies,or other neurodegenerative conditions, as well as other neuralconditions such as dystonia.

For instance, in the case of Parkinson's Disease, the wearable datacollection device 1104 may be configured to collect data, using thesmall motion analysis algorithm 1110 m and/or other algorithms 1110,related to rhythmic, side-to side and rotational head motions that arecharacteristic of the condition. Further, the learning engines 1118corresponding to the Parkinson's Disease condition may apply patternanalysis and/or other analysis to identify variance(s) in those motionscorresponding to data capture-related metadata such as, in someexamples, time of day of data capture, location at time of capture, etc.Further, the learning engines 1118 may correlate collected data tosubject clinical data, such as contemporaneous medical interventionsand/or medication schedule (e.g., accessed from a separate system and/oridentified by the prescription intake identifying algorithm 1110 n). Inan additional example, the learning engines 1118 may correlate smallmotion data with data obtained through other algorithms 1110 such as, insome examples, diet data collected by the dietary intake identifier 1110f, activity data collected by the activity identifier 1110 d, mentaltasks and engagement cues collected, for example, by the fatigueanalysis algorithm 1110 e, eye motion analysis algorithm 1110 c, and/orvocalization analysis algorithm 1110 o, and/or environmental conditionsand events collected by the noise intensity analysis algorithm 1110 j,event identifier 1110 a, and/or scent analysis algorithm 1110 g.Further, additional algorithms 1110 and/or external data may providecyclical fluctuation data such as circadian rhythms and/or seasonalrhythms for correlation with the small motion data by the learningengines 1118. Although described in relation to the various algorithms1110, in other implementations, data may be accessed form a separatesystem (e.g., such as a patient information portal connecting thelearning engines 1118 to user medical records), input directly by thewearer, and/or input to an independent software application accessed bya caregiver, physician, or other individual.

In some implementations, small motion data collected by the wearabledata collection device 1104 (e.g., via algorithms such as the smallmotion analysis algorithm 1110 m, eye motion analysis algorithm 1110 c,heat fluctuation analysis algorithm 1110 l, and/or small noise analysisalgorithm 1110 p) may be used to assist in diagnosis of an unwellnesscondition such as Parkinson's. For example, a practitioner may employthe wearable data collection device 1104 as a tool for gatheringinformation regarding an individual outside of a clinician's office. Theindividual, for example, may be instructed to don the wearable datacollection device 1104 for a certain period of time to provide data tothe practitioner in identifying an unwellness condition orstage/progression of the unwellness condition. The learning engines 1118may include a diagnosis support module configured to identifysimilarities between data patterns collected by the wearable datacollection device 1104 and patterns associated with one or moreunwellness conditions and provide this information to the practitionerfor analysis. Additionally, data collected may be “crowd sourced” andanalyzed to refine small motion recognition patterns for behaviorsrelated to an unwellness condition such as Parkinson's as well as smallmotion recognition patterns matching particular stages or progressionsof a particular unwellness condition. In a particular example, patternanalysis may be used to identify small motions indicating an imminentseizure episode in individuals with epilepsy.

In some implementations, as an ongoing support tool for practitionermonitoring of an individual diagnosed with an unwellness condition, thepractitioner may review data collected by the wearable data collectiondevice 1104 for periodic evaluations or check-ups, for example to tracksymptoms, symptom severity, and/or frequency of symptomatic behaviors.Additionally, with the support of data collected by other algorithms1110, the practitioner may be presented with patterns identified by thelearning engines 1118 related to controlled and non-controlled factorstrending to correlate with the expression of symptoms or with symptomseverity.

In some implementations, the individual 1102 uses the wearable datacollection device 1104 in an ongoing manner to aid in managing symptomsand/or evaluating interventions or treatments related to behaviorsidentified through the algorithms 1110. The individual 1102, in aparticular example, may wear the wearable data collection device 1104 aspart of a clinical trial related to a particular treatment orintervention for an unwellness condition. In another example, thewearable data collection device 1104 may be configured to providefeedback directly to the individual 1102 to support management ofsymptoms. In either of the above cases, the learning engines mayidentify patterns of behaviors correlating to elements within directcontrol of the individual 1102 which appear to contribute to thefrequency or severity of symptoms and recommend non-clinicalinterventions that the individual 1102 can personally attempt to managethe unwellness condition. The behaviors, in some examples, may includediet, meditation, exercise, sleep patterns, or ingestion of stimulants.

In some implementations, the wearable data collection device 1104 mayprovide cues for immediate management of symptoms or behaviorscorresponding to an unwellness condition. For example, the learningengines 1118 may use the data 1114 related to small (e.g., head) motionsand their dynamics to make ongoing assessments or quantifications of thesymptoms and behaviors of the individual 1102 and feed back learned data1124, such as volitional control or biofeedback data, for use inempowering the individual 1102 to conduct “smart management” of symptomsor behaviors, thus gaining better control and autonomy. The feedback,for example, may be presented to the individual 1102 via the wearabledata collection device 1104 or another peripheral computing device toprovide cues to the individual 1102 for suppressing or extinguishingsymptoms or behaviors. In a particular example for an unwellnesscondition involving vestibular system damage, leading to loss ofbalance, based upon how level the individual 1102 is maintaining headposition, the wearable data collection device 1104 may prompt theindividual 1102 (e.g., with a visual target on a heads-up display) toadjust head positioning. Further to this example, the wearable datacollection device 1104 may include a balance coaching module fortraining the individual 1102 to accurately compensate for the effects ofthe vestibular system damage through correction and feedback. Similarmanagement techniques may be applied an individual 1102 withHuntington's Disease to support the individual 1102 in management ofstereotypical Huntington's Chorea movements. In another illustration,the system 1100 may analyze small motion data 1114 to anticipate onsetof a seizure in an epileptic individual 1102. In anticipation of seizureactivity, the system 1100 may issue a warning to the individual 1102 viathe wearable data collection device 1104 or other peripheral computingdevice.

In some implementations, feedback may incorporate suggestions of copingmechanisms for coping with behavioral episodes stemming from aparticular unwellness condition, such as, in some examples, panicdisorders and attention deficit hyperactivity disorder (ADHD). Thewearable data collection device 1104, in a particular example, mayvisually present and/or “whisper” an attention focusing mechanism for anindividual 1102 coping with ADHD to perform to regain focus. The system1100, further, may monitor and assess effectiveness of a given copingmechanism for the particular individual 1102, such as a deep breathingexercise for controlling panic.

Rather than or in addition to feeding information back to the individual1102, in some implementations, the learning engines 118 may generatelearned data 1124 for use by one or more systems within or incommunication with the wearable data collection device 1104 and/or theindividual 1102 to support automated or semi-automated interventions.Such interventions may include, but are not limited to, triggering animplanted device that can disseminate drugs into the body of theindividual 1102 appropriately to treat the symptoms or mechanisms of theunwellness condition (e.g., injecting L-Dopa or related pharmaceuticalsinto the body, etc.) or triggering a neural stimulation device such as adeep brain electrical stimulator or a stimulator using transcranialmagnetic or direct-current stimulation.

In a semi-automated intervention, rather than triggering a therapeuticresponse to identified symptoms, the wearable data collection device1104 may prompt the individual 1102 for approval of the intervention.For example, a message may appear on a heads-up display of the wearabledata collection device 1104, requesting approval to proceed with anidentified intervention. In another example, rather than prompting forapproval of the individual 1102, the system 1100 may prompt a caregiveror practitioner for authorization to exercise the intervention.Combinations of these features are possible. For example, based upon theperceived immediacy and criticality of the intervention, the system 1100may exercise an automatic intervention rather than a semi-automaticintervention (e.g., in the circumstance where the system 1100anticipates that the individual 1102 is not in a condition to provideapproval).

In the event of a serious condition needing intervention, in someimplementations, the system 1100 may present a medical alert to medicalprofessionals, such as calling for an ambulance or directing a medic ata treatment facility to the current location of the individual 1102. Thewearable data collection device 1104, for example, may derivecoordinates (e.g., GPS coordinates, an address, etc.) for directing aidto the individual 1102. If the medical professionals addressed areconnected to the system 1100 (e.g., via a coordinating softwareapplication, etc.), the system 1100 may provide a feed of data and otherinformation for immediate assessment of the condition, such as a portionof the data and analysis information 1114 most recently and/or currentlycaptured. In another example, if the system 1100 has a directcommunication link with the medical professionals (e.g., telephonenumber for text message or short recorded message), the system 1100 mayissue a message to the medical professionals with brief assessment data.

In some implementations, the algorithms 1110, individually, in concert,or through data review provided by one or more learning engines 1118,may provide information to a video and/or gaming system to assess theindividual's response to a video or game presented to the individual1102. The video or gaming system may be part of the wearable datacollection device 1104 or another computing system in communication withthe system 1100. In a particular example, a marketing algorithm mayassess the individual's response to the video or game to identify oranticipate the individual's interest in material such as advertisements,political campaign materials, products, product marketing, or othermaterials involving personal preferences and/or having commercialinterests. In another example, a simulation or training system mayinclude one or more algorithms for assessing responses to participantsof a simulation (e.g., military training, police officer training,flight training, etc.), such as emotional response.

In some implementations, the video or gaming system may use theassessment of the response of the individual 1102 to the video or gameto influence the structure of a game or video that the individual 1102is presently engaged in. For example, data derived from the algorithms1110 may be used to alter a difficulty level, direction, or mode of thevideo game to enhance a desired response from the individual 1102. In aparticular example, if the individual 1102 appears bored ordisinterested, the difficulty, direction, and/or mode of the game may bealtered to encourage great interest from the individual 1102. In anotherexample, the video or gaming system may identify responses ofexcitement, fear, or other arousal and, in response, provide additionalvideo or game sequences which are similar in nature (e.g., anticipatedto elicit the same or similar response from the individual 1102).

In some implementations, the algorithms 1110, individually, in concert,or through data review provided by one or more learning engines 1118,provide feedback 1112 regarding inclination towards an impending adversehealth event or atypical behavioral episode. For example, depending uponthe severity and/or certainty of the impending adverse health event, theindividual 1102, a caregiver, and/or a physician may be alerted to theimpending health concern. For example, the wearable data collectiondevice donned by the individual 1102 may present an audible and/orvisual warning regarding the likelihood of an impending health event oratypical behavioral episode and, potentially, an indication of the typeof event anticipated. Furthermore, the individual 1102 may be promptedwith recommendations of measures to take to best prevent, redirect,and/or minimize the atypical behavioral episode (e.g., take an aspirin).The subject, in some implementations, may be presented with feedback1112 designed to divert a pending health event. For example, feedback1112 may be presented via the subject's wearable data collection device1104 (e.g., visual, audible, tactile, etc. feedback) designed to alterone or more physiological conditions indicative of a pending healthevent, such as subduing a panic attack.

In some implementations, the learning engines 1118 evaluates eventsidentified by the event identifier 1110 a associated with manyindividuals as well as corresponding metadata (e.g., demographics,geographic location, time, weather patterns, and other aspectsassociated with the onset of the event) to identify event patternssimilar to a subject group. In some examples, the learning engines 1118may identify a particular location at a particular time of dayassociated with multiple events, such as Tuesdays at 12:00 at aparticular intersection of a downtown area. Further, the learningengines 1118 may recognize, from archive data 1122, that the events areall associated with a loud noise. For example, a train may pass nearbythe intersection on one or more days of the week at particular times,and the whistle of the train may trigger events in one or moreindividuals susceptible to loud noises. In identifying geographic (and,optionally temporal) “hot spots”, the system 1100 may further evolve thecapability of issuing warnings to other individuals (or caregiversthereof) within the suspect geographic area at a suspect time.

Further, in some implementations the learning engines 1118 analyze eventdata corresponding to a collection of individuals to generate a hot spotmap. The hot spot map, for example, may be supplied to researchers andclinicians for further review and analysis. In another example, the hotspot map may be supplied to individuals and/or caregivers forinformational purposes. As the learning engines 1118 evolve in analysisof event data, the hot spot map may be refined to maps corresponding toindividuals having similar demographic, diagnostic, and/or clinicalbackgrounds. For example, a PTSD hot spot map may differ from a ASD hotspot map.

Although described above as learning algorithms 1118, in otherimplementations, a portion or all of the learning algorithms 1118 may bereplaced with assessment algorithms 1118 lacking an adaptive learningcapability. For example, static algorithms for analyzing the data andanalysis information 1114 may perform similar roles to learningalgorithms 1118 but are not learning algorithms in that they do notchange or evolve relative to new data. Instead, static algorithms may bedesigned to filter or extract information from the data and analysisinformation 1114, transform, analyze, and/or combine data 1114 withexternally obtained data to perform various functions described abovewhile remaining stable over time until they are altered, updated, orreplaced. As with the learning engines 1118, one or more staticalgorithms may be programmed initially into the software, firmware,and/or hardware of a component of the wearable data collection device1104 or other peripheral computing system. As with the learning engines1118, static algorithms may also be updated from time to time, forinstance in the process of updating software or firmware or hardware asmay be accomplished, in some examples, via remote-pushed updates, byuser intervention, or by servicing by service technicians.

In some implementations, one or more of the learning algorithms 1118 arereplaced or enhanced by concierge intervention via a conciergeintervention system (not illustrated) including a data connection to oneor more computer systems, such as a network portal connection, to supplydata and analysis information 1114 and/or data, analysis, and learninginformation 1120 to a human operator. In this manner, the conciergeintervention system may be used in a manner whereby data related to theindividual 1102 may be processed in part by human operators, including,for example, trained health practitioners, data analysts, and/ortechnicians, rather than being processed solely by automated processes(e.g., algorithms 1110 and/or learning engines 1118). The humanoperator, for example, may review the data and analysis information 1114and/or data, analysis, and learning information 1120, performing actionsand mental tasks that replace or augment one or more functions or rolesperformed by learning algorithms 1118. During review of the data andanalysis information 1114 and/or data, analysis, and learninginformation 1120, the actions and mental tasks performed by a humanoperator may involve or be supplemented by actions or datatransformations executing upon a computing device. In one illustrativeexample, a human operator may review data obtained by the small motionanalysis algorithm 1110 m to manually count heart beats or breaths,potentially with the assistance of some analysis or computationsoftware. The human operator may further enter results of the manualcount into the computing device to feed the information back into thesystem 1100. In another illustrative example, the concierge interventionsystem can receive the voice recording data 116 a collected by thewearable data collection device 1104. In such as example, a humanoperator may listen to the voice recording data 116 a, count the breathsbased on the sound of the person breathing in and out, and then forwardthe results of this analysis (e.g., manual breath count) to the system1100 (e.g., the learning engines 1118, wearable data collection device1104, archive data 1122, etc.). In some implementations, the conciergeintervention system may perform the same or similar functions performedby the learning algorithms 1118 and/or algorithms 1110, for instance incases of quality assurance or oversight or during testing.

In another example, feedback 1112 may be designed to correct for anissue exhibited by the individual 1102. For example, based upon analysisof vestibular dynamics data 1106 f, feedback 1112 regarding presentbalance may be presented to the individual 1102. Further, a game or andtask such as virtual balance beam may be presented to the individual1102 to encourage corrective behavior.

In some implementations, a subject identification algorithm 1110 h mayreview the data 1108 or analysis information derived by one or more ofthe other algorithms 1110 to uniquely identify the individual 1102 basedupon biometric identification. The biometric identification, in turn,may be used to recognize a current user of the data collection device1104 in view of a group of potential users (e.g., family members, healthclub members, etc.). Furthermore, the biometric identification may beused in an authentication process when communicating with third partysystems via the data collection device 1104 such as, in some examples,web sites, banks, ATMs, or building security access systems.

The learning engines 1118, in some implementations, review the data 1108and analysis information 1114 for biometric signatures regarding groupsof individuals. For example, biometric similarities may be derived infamilies, age groups, racial classifications, and/or disease categories.

Next, a hardware description of an example wearable data collectiondevice according to exemplary embodiments is described with reference toFIG. 12. In FIG. 12, the wearable data collection device includes a CPU1200 which performs a portion of the processes described above. Theprocess data and instructions may be stored in memory 1202. Theseprocesses and instructions may also be stored on a storage medium disk1204 such as a portable storage medium or may be stored remotely.Further, the claimed advancements are not limited by the form of thecomputer-readable media on which the instructions of the inventiveprocess are stored. For example, the instructions may be stored in FLASHmemory, RAM, ROM, or any other information processing device with whichthe wearable computing system communicates, such as a server orcomputer.

Further, components of the claimed advancements may be provided as autility application, background daemon, or component of an operatingsystem, or combination thereof, executing in conjunction with CPU 1200and an operating system such as and other systems known to those skilledin the art.

CPU 1200 may be an ARM processor, system-on-a-chip (SOC),microprocessor, microcontroller, digital signal processor (DSP), or maybe other processor types that would be recognized by one of ordinaryskill in the art. Further, CPU 1200 may be implemented as multipleprocessors cooperatively working in parallel to perform the instructionsof the inventive processes described above.

The wearable computing system in FIG. 12 also includes a networkcontroller 1206 for interfacing with network 1228. As can beappreciated, the network 1228 can be a public network, such as theInternet, or a private network such as an LAN or WAN network, or anycombination thereof and can also include PSTN or ISDN sub-networks. Thenetwork 1228 can be wireless such as a cellular network including EDGE,3G and 4G wireless cellular systems. The wireless network can also beWi-Fi, Bluetooth, or any other wireless form of communication that isknown.

The wearable data collection device further includes a displaycontroller 1208 interfacing with display 1210, such as a remotelylocated display or a heads up display. A general purpose I/O interface1212 interfaces with an input device (e.g., microphone for voicecommands, etc.). General purpose I/O interface can also communicate witha variety of on board I/O devices 1216 and/or peripheral I/O devices1218 including, in some examples, a video recording system, audiorecording system, microphone, gyroscopes, accelerometers, gravitysensors, linear accelerometers, global positioning system,magnetometers, EEG, EMG, EKG, bar code scanner, QR code scanner, RFIDscanner, temperature monitor, skin dynamics sensors, scent monitor,light monitor, blood dynamics and chemistry monitor, vestibular dynamicsmonitor, external storage devices, and external speaker systems.

A sound controller 1220 is also provided in the wearable data collectiondevice, to interface with speakers/microphone 1222 thereby bothrecording and presenting sounds to the wearer.

The general purpose storage controller 1224 connects the storage mediumdisk 1204 with communication bus 1226, such as a parallel bus or aserial bus such as a Universal Serial Bus (USB), or similar, forinterconnecting all of the components of the wearable computing system.A description of the general features and functionality of the display1210, as well as the display controller 1208, storage controller 1224,network controller 1206, sound controller 1220, and general purpose I/Ointerface 1212 is omitted herein for brevity as these features areknown.

The wearable data collection device in FIG. 12, in some embodiments,includes a sensor interface 1230 configured to communicate with one ormore onboard sensors 1232 and/or one or more peripheral sensors 1234.The onboard sensors 1232, for example, can be incorporated directly intothe internal electronics and/or a housing of the wearable device. Theperipheral sensors 1234 can be in direct physical contact with thesensor interface 1230 e.g. via a wire; or in wireless contact e.g. via aBluetooth, Wi-Fi or NFC connection. Alternatively, one or more of theperipheral sensors 1234 may communicate with the sensor interface 1230via conduction through the body tissue or via other mechanisms.Furthermore, one or more peripheral sensors 1234 may be in indirectcontact e.g. via intermediary servers or storage devices that are basedin the network 1228; or in (wired, wireless or indirect) contact with asignal accumulator somewhere on or off the body, which in turn is in(wired or wireless or indirect) contact with the sensor interface 1230.The peripheral sensors 1234 can be arranged in various types ofconfigurations relative to the body. For instance, they can be mountedon the body, near the body, looking at the body, and/or implanted withinthe body of a human or animal subject. The onboard sensors 1232 and/orperipheral sensors 1234 can include, in some examples, one or moremicrophones, bone-conduction microphones, physiological eventsmicrophones, cameras, video cameras, high-speed cameras, temperaturemonitors, accelerometers, gyroscopes, magnetic field sensors, magneticcompasses, tap sensors and/or vibration sensors—internal or external toa gyroscope/accelerometer complex, infrared sensors or cameras, and/oreye-tracking cameras or eye-tracking sensor complex. In furtherexamples, onboard sensors 1232 and/or peripheral sensors 1234 mayinclude one or more skin-mounted electrodes, body-proximal electrodes(contact or non-contact), pulse oximetry devices, laser and laser-lightsensors, photodiodes, galvanic skin response sensor modules, RF or otherelectromagnetic signal detectors, electrical signal pre-amplifiers,electrical signal amplifiers, electrical signal hardware filter devices,chemical sensors, and/or artificial noses.

A group of sensors communicating with the sensor interface 1230 may beused in combination to gather a given signal type from multiple placessuch as in the case of EEG or skin temperature in order to generate amore complete map of signals. One or more sensors communicating with thesensor interface 1230 can be used as a comparator or verificationelement, for example to filter, cancel, or reject other signals. Forinstance, a light sensor can pick up ambient light or color changes anduse them to subtract or otherwise correct light-based signals from acamera pointed at the eye or skin to pick up small color or reflectancechanges related to physiological events. Likewise, a microphone mountedagainst the body can pick up internal sounds and the voice of thesubject donning the wearable data communication device and subtract theinternal sounds from ambient sounds such as the voice of a separateindividual or noise from environmental events, in order to moreconcentrate on the audible features of external events. Conversely,sensor data may be used to subtract environmental noise frombody-internal sound signatures that can give evidence of physiology.Similarly, the input of multiple temperature monitors can aid inadjusting for major changes in ambient temperature or for narrowing atemperature signature to more narrowly identify the temperature of aparticular element (e.g., device/electronics temperature or bodytemperature) without contamination from heat provided by other elements.

The wearable data collection device in FIG. 12, in some embodiments,includes a stimulation interface 1236 for supplying stimulation feedbackto an individual donning the wearable data collection device. Thestimulation interface 1236 is in communication with one or more onboardstimulators 1238 and/or peripheral stimulators 1240 configured todeliver electrical pulses to the individual, thereby alteringphysiological conditions of the individual. For example, one or moreonboard stimulators 1238 and/or peripheral stimulators 1240 may besituated and/or configured to electrically stimulate heart rate orbreathing or brain waves at particular frequencies. The onboardstimulators 1238 and/or peripheral stimulators 1240 can be mounted on ornear the body, and/or implanted within the body, and can includecomponents that are external and others that are internal to the bodywhich may be configured for intercommunication with each other. In someexamples, onboard stimulators 1238 and/or peripheral stimulators 1240can include one or more of electrical signal generators and stimulation(output) electrodes, vibrator devices, heat-imparting devices,heat-extraction devices, sound generators/speakers, electromagnets,lasers, LEDs and other light sources, drug administering devices, brainstimulation or neural stimulation devices, gene transcription orexpression modulation system, and/or pain or sensory stimulationgenerators.

Next, a hardware description of the computing device, mobile computingdevice, or server according to exemplary embodiments is described withreference to FIG. 13. In FIG. 13, the computing device, mobile computingdevice, or server includes a CPU 1300 which performs the processesdescribed above. The process data and instructions may be stored inmemory 1302. These processes and instructions may also be stored on astorage medium disk 1304 such as a hard drive (HDD) or portable storagemedium or may be stored remotely. Further, the claimed advancements arenot limited by the form of the computer-readable media on which theinstructions of the inventive process are stored. For example, theinstructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM,PROM, EPROM, EEPROM, hard disk or any other information processingdevice with which the computing device, mobile computing device, orserver communicates, such as a server or computer.

Further, a portion of the claimed advancements may be provided as autility application, background daemon, or component of an operatingsystem, or combination thereof, executing in conjunction with CPU 1300and an operating system such as Microsoft Windows 7, UNIX, Solaris,LINUX, Apple MAC-OS and other systems known to those skilled in the art.

CPU 1300 may be a Xenon or Core processor from Intel of America or anOpteron processor from AMD of America, or may be other processor typesthat would be recognized by one of ordinary skill in the art.Alternatively, the CPU 1300 may be implemented on an FPGA, ASIC, PLD orusing discrete logic circuits, as one of ordinary skill in the art wouldrecognize. Further, CPU 1300 may be implemented as multiple processorscooperatively working in parallel to perform the instructions of theinventive processes described above.

The computing device, mobile computing device, or server in FIG. 13 alsoincludes a network controller 1306, such as an Intel Ethernet PROnetwork interface card from Intel Corporation of America, forinterfacing with network 13X. As can be appreciated, the network 1328can be a public network, such as the Internet, or a private network suchas an LAN or WAN network, or any combination thereof and can alsoinclude PSTN or ISDN sub-networks. The network 1328 can also be wired,such as an Ethernet network, or can be wireless such as a cellularnetwork including EDGE, 3G and 4G wireless cellular systems. Thewireless network can also be Wi-Fi, Bluetooth, or any other wirelessform of communication that is known.

The computing device, mobile computing device, or server furtherincludes a display controller 1308, such as a NVIDIA GeForce GTX orQuadro graphics adaptor from NVIDIA Corporation of America forinterfacing with display 1310, such as a Hewlett Packard HPL2445w LCDmonitor. A general purpose I/O interface 1312 interfaces with a keyboardand/or mouse 1314 as well as a touch screen panel 1316 on or separatefrom display 1310. General purpose I/O interface also connects to avariety of peripherals 1318 including printers and scanners, such as anOfficeJet or DeskJet from Hewlett Packard.

A sound controller 1320 is also provided in the computing device, mobilecomputing device, or server, such as Sound Blaster X-Fi Titanium fromCreative, to interface with speakers/microphone 1322 thereby providingsounds and/or music.

The general purpose storage controller 1324 connects the storage mediumdisk 1304 with communication bus 1326, which may be an ISA, EISA, VESA,PCI, or similar, for interconnecting all of the components of thecomputing device, mobile computing device, or server. A description ofthe general features and functionality of the display 1310, keyboardand/or mouse 1314, as well as the display controller 1308, storagecontroller 1324, network controller 1306, sound controller 1320, andgeneral purpose I/O interface 1312 is omitted herein for brevity asthese features are known.

One or more processors can be utilized to implement various functionsand/or algorithms described herein, unless explicitly stated otherwise.Additionally, any functions and/or algorithms described herein, unlessexplicitly stated otherwise, can be performed upon one or more virtualprocessors, for example on one or more physical computing systems suchas a computer farm or a cloud drive.

Reference has been made to flowchart illustrations and block diagrams ofmethods, systems and computer program products according toimplementations of this disclosure. Aspects thereof are implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of this disclosure. For example, preferableresults may be achieved if the steps of the disclosed techniques wereperformed in a different sequence, if components in the disclosedsystems were combined in a different manner, or if the components werereplaced or supplemented by other components. The functions, processesand algorithms described herein may be performed in hardware or softwareexecuted by hardware, including computer processors and/or programmablecircuits configured to execute program code and/or computer instructionsto execute the functions, processes and algorithms described herein.Additionally, some implementations may be performed on modules orhardware not identical to those described. Accordingly, otherimplementations are within the scope that may be claimed.

The invention claimed is:
 1. A system for providing monitoring of andanalysis related to a wearer of a wearable data collection device, thesystem comprising: a wearable data collection device comprisingprocessing circuitry, and a non-transitory computer readable mediumhaving instructions stored thereon; and one or more input captureelements connected to and/or in communication with the wearable datacollection device; wherein the instructions, when executed by theprocessing circuitry of the wearable data collection device, cause theprocessing circuitry to: collect, via at least one of the one or moreinput capture elements, sensor data, wherein the sensor data includes atleast one of image data, audio data, and motion data, analyze the sensordata to identify at least one socially relevant behavior of the wearerthat is responsive to at least one socially relevant event involving aninteraction between the wearer and at least one other participant,wherein the at least one socially relevant behavior includes a gestureof the wearer, and identifying the at least one socially relevantbehavior comprises determining a threshold difference between the atleast one socially relevant behavior of the wearer and a conventionalresponse to the at least one socially relevant event, wherein the atleast one socially relevant event occurs prior to or concurrently withthe at least one socially relevant behavior, using the thresholddifference, identify whether the behavior of the wearer varies from theconventional response, store, upon a non-transitory computer readablestorage device, information related to the at least one sociallyrelevant behavior, responsive to identifying that the behavior of thewearer varies from the conventional response, present, to the wearer viaone or more output features of the wearable data collection device,feedback directed to modifying the socially relevant behavior, analyzesensor data captured subsequently to a timeframe of presenting thefeedback to identify a subsequent gesture of the wearer responsive tothe feedback, determine whether the subsequent gesture more closelymatches the conventional response, and present, via at least one outputfeature of the one or more output features of the wearable datacollection device, subsequent feedback to the wearer, wherein thesubsequent feedback comprises at least one of i) indication of successor failure in performing the conventional response, and ii) progressfeedback comprising at least one of a) an ongoing score, b) acomparative score comparing a prior session to a present session, c) acompetitive score comparing a score of the wearer to a score of one ormore additional individuals, and d) a level completion status.
 2. Thesystem of claim 1, wherein the information related to the at least onesocially relevant behavior comprises at least one of a verbal repetitioncount and a movement repetition count.
 3. The system of claim 1, whereinthe conventional response is based at least in part upon statisticalanalysis of at least one of i) a spectrum of historical responsesidentified as being common to the wearer, and ii) a personalityassessment of the wearer.
 4. The system of claim 1, wherein theinstructions, when executed by the processing circuitry, cause theprocessing circuitry to: determine the conventional response based atleast in part on one or more mitigating factors.
 5. The system of claim1, wherein the instructions, when executed by the processing circuitry,cause the processing circuitry to provide, to a third party computingdevice, feedback related to the socially relevant behavior, wherein:providing feedback to the third party computing device comprisestransmitting, via a wired or wireless transmission link, a datatransmission to the third party computing device identifying thesocially relevant behavior.
 6. The system of claim 1, wherein: providingfeedback to the wearer comprises providing coaching feedback to thewearer for modifying the behavior of the wearer to more closely matchthe conventional response; and the conventional response comprises atleast one of i) suppression of the gesture, and ii) calming of anemotional state of the wearer.
 7. The system of claim 5, whereinproviding feedback to the third party computing device comprisesforwarding, to the third party computing device, at least one of a) aportion of the sensor data, and b) data derived from analyzing thesensor data for review by an evaluator.
 8. The system of claim 1,wherein providing feedback to the wearer comprises providing, via atleast one output feature of the one or more output features of thewearable data collection device, at least one of an audible cue and avisual cue directing the wearer to perform the conventional response. 9.The system of claim 1, further comprising a remote processing systemaccessible to the wearable data collection device via a network, whereinanalyzing the sensor data comprises: providing, via the network to theremote processing system, at least a portion of the sensor datacontemporaneous with a timeframe of the socially relevant behavior ascontext data; and receiving, via the network from the remote processingsystem, an indication of the at least one socially relevant behaviorvarying from the conventional response.
 10. The system of claim 1,wherein: the sensor data comprises physiological motion data; and aphysiological state of the wearer is identified based upon at least oneof heart dynamics and breathing dynamics.
 11. The system of claim 1,wherein: the conventional response comprises direction of gaze of thewearer towards a first participant of the at least one participant; andthe instructions, when executed by the processing circuitry, cause theprocessing circuitry to present, upon a heads up display of the wearabledata collection device, an augmented version of a real-time videostream, wherein presenting the feedback directed to modifying thesocially relevant behavior comprises visually augmenting the real-timevideo stream in an area upon and/or surrounding a face region of thefirst participant to draw attention to the face region of the firstparticipant responsive to the at least one socially relevant behaviorincluding failure to direct gaze toward the first participant.
 12. Thesystem of claim 11, wherein: analyzing the sensor data capturedsubsequently to the timeframe of presenting the feedback comprisesobtaining, from the sensor data, data indicative of a direction of gazeof the wearer; and determining whether the subsequent gesture moreclosely matches the conventional response comprises determining, usingthe data indicative of the direction of gaze of the wearer,co-registration between the direction of gaze of the wearer and the faceregion of the first participant.
 13. The system of claim 1, furthercomprising, responsive to determining whether the subsequent reactionmore closely matches the conventional response: gauging effectiveness ofthe feedback based at least in part on relative change in behavior ofthe wearer toward the conventional response; and storing, within anon-transitory computer readable medium connected to or in communicationwith the wearable data collection device, an indication of theeffectiveness of the feedback, wherein providing the feedback comprisesproviding one or more types of feedback having a respective indicationof effectiveness indicative of being previously effective in obtainingthe conventional response from the wearer.
 14. The system of claim 13,wherein the instructions, when executed by the processing circuitry ofthe wearable data collection device, cause the processing circuitry to,prior to presenting the feedback: select, from a plurality of feedbackoptions based at least in part upon respective archival indications ofeffectiveness associated with one or more feedback options of theplurality of feedback options, the feedback.
 15. The system of claim 4,wherein the one or more mitigating factors include at least one of anemotional state of the participant prior to the socially relevant event,and a physiological state of the wearer prior to the socially relevantevent.
 16. A method for assisting a wearer of a wearable data collectiondevice in responding to socially relevant events, the system comprising:collecting, via at least one of one or more input capture elementsconnected to and/or in communication with the wearable data collectiondevice, sensor data, wherein the sensor data includes at least one ofimage data, audio data, and motion data; analyzing, via processingcircuitry, the sensor data to identify a gesture made by the wearer inresponse to at least one socially relevant event involving aninteraction between the wearer and at least one other participant,wherein the at least one socially relevant event occurs prior to orconcurrently with the gesture; identifying, by the processing circuitry,at least one conventional response to the at least one socially relevantevent; determining, by the processing circuitry, a measured differencebetween the gesture and the at least one conventional response; usingthe measured difference, determining, by the processing circuitry,whether the gesture of the wearer varies from the conventional responseby at least a threshold amount; storing, upon a non-transitory computerreadable storage device, information related to the gesture; responsiveto determining that the gesture of the wearer varies from theconventional response by at least a threshold amount, selecting, by theprocessing circuitry based at least in part on the gesture, feedbackdirected to modifying or encouraging a socially relevant behavior of thewearer to produce a subsequent conventional response, presenting, to thewearer via one or more output features of the wearable data collectiondevice, the feedback, analyzing, by the processing circuitry, sensordata captured subsequently to a timeframe of presenting the feedback toidentify a subsequent gesture of the wearer responsive to the feedback,determining, by the processing circuitry, whether the subsequent gesturecorresponds to the subsequent conventional response, and presenting, viaat least one output feature of the one or more output features of thewearable data collection device, subsequent feedback to the wearer,wherein the subsequent feedback comprises at least one of i) indicationof success or failure in performing the subsequent conventionalresponse, and ii) progress feedback comprising at least one of a) anongoing score, b) a comparative score comparing a prior session to apresent session, c) a competitive score comparing a score of the wearerto a score of one or more additional individuals, and d) a levelcompletion status.
 17. The system of claim 1, wherein providing feedbackto the wearer comprises providing, via at least one output feature ofone or more output features of the wearable data collection device, atleast one of visual, audible, haptic, and neural stimulation feedback tothe wearer.
 18. The system of claim 5, wherein providing feedbackcomprises identifying one or more recommended measures for at least oneof the wearer and a third party to take to prevent, redirect, and/orminimize the socially relevant behavior.